[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:46:39 dist-test-slave-w3bm krb5kdc[3967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:46:39 dist-test-slave-w3bm krb5kdc[3967](info): set up 2 sockets
Sep 28 17:46:39 dist-test-slave-w3bm krb5kdc[3967](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 17:46:40.611393 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:46:40 dist-test-slave-w3bm krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:46:40.725502 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:43419
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:43419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:40.802205 3980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:40.802389 3980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:40.802408 3980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:40.803529 3980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:40.803566 3980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:40.803579 3980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:40.803591 3980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:40.803615 3980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:40.805255 3980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:43419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:43419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:40.805505 3980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:40.805729 3980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:46:40.811030 3980 server_base.cc:1034] running on GCE node
W20260928 17:46:40.811184 3986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:40.811268 3988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:40.811197 3985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:40.811553 3980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:40.811837 3980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:40.812999 3980 hybrid_clock.cc:648] HybridClock initialized: now 1790617600812981 us; error 31 us; skew 500 ppm
Sep 28 17:46:40 dist-test-slave-w3bm krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617600, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:40.815286 3980 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:46:40.815744 3980 webserver.cc:460] Webserver started at http://127.3.222.62:37175/ using document root <none> and password file <none>
I20260928 17:46:40.815951 3980 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:40.815996 3980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:40.816111 3980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:40.817111 3980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "334e8580fc2c400c86f6f4a1c362f538"
format_stamp: "Formatted at 2026-09-28 17:46:40 on dist-test-slave-w3bm"
server_key: "ffa3c029f440972a537261272d4508a7"
server_key_iv: "7547fcaa06956f72ee668f2a47c78d16"
server_key_version: "encryptionkey@0"
I20260928 17:46:40.817436 3980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "334e8580fc2c400c86f6f4a1c362f538"
format_stamp: "Formatted at 2026-09-28 17:46:40 on dist-test-slave-w3bm"
server_key: "ffa3c029f440972a537261272d4508a7"
server_key_iv: "7547fcaa06956f72ee668f2a47c78d16"
server_key_version: "encryptionkey@0"
I20260928 17:46:40.818658 3980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:46:40.819399 3995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:40.819550 3980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:40.819628 3980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "334e8580fc2c400c86f6f4a1c362f538"
format_stamp: "Formatted at 2026-09-28 17:46:40 on dist-test-slave-w3bm"
server_key: "ffa3c029f440972a537261272d4508a7"
server_key_iv: "7547fcaa06956f72ee668f2a47c78d16"
server_key_version: "encryptionkey@0"
I20260928 17:46:40.819686 3980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:40.833834 3980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:40.835618 3980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:40.835755 3980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:40.839249 3980 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:43419
I20260928 17:46:40.839280 4047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:43419 every 8 connection(s)
I20260928 17:46:40.839602 3980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:40.840389 4048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:40.842537 4048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538: Bootstrap starting.
I20260928 17:46:40.843071 4048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:40.843340 4048 log.cc:824] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:40.843966 4048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538: No bootstrap required, opened a new log
I20260928 17:46:40.845769 4048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e8580fc2c400c86f6f4a1c362f538" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43419 } }
I20260928 17:46:40.845921 4048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:40.845945 4048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 334e8580fc2c400c86f6f4a1c362f538, State: Initialized, Role: FOLLOWER
I20260928 17:46:40.846071 4048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e8580fc2c400c86f6f4a1c362f538" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43419 } }
I20260928 17:46:40.846138 4048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:40.846160 4048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:40.846206 4048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:40.846721 4048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e8580fc2c400c86f6f4a1c362f538" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43419 } }
I20260928 17:46:40.846802 4048 leader_election.cc:302] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 334e8580fc2c400c86f6f4a1c362f538; no voters:
I20260928 17:46:40.846946 4048 leader_election.cc:288] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:40.846985 4051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:40.847154 4051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [term 1 LEADER]: Becoming Leader. State: Replica: 334e8580fc2c400c86f6f4a1c362f538, State: Running, Role: LEADER
I20260928 17:46:40.847172 4048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:40.847265 4051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e8580fc2c400c86f6f4a1c362f538" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43419 } }
I20260928 17:46:40.847625 4053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 334e8580fc2c400c86f6f4a1c362f538. Latest consensus state: current_term: 1 leader_uuid: "334e8580fc2c400c86f6f4a1c362f538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e8580fc2c400c86f6f4a1c362f538" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43419 } } }
I20260928 17:46:40.847704 4053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:40.847746 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "334e8580fc2c400c86f6f4a1c362f538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e8580fc2c400c86f6f4a1c362f538" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43419 } } }
I20260928 17:46:40.847820 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:40.848383 4060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:40.848839 4060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:46:40.849799 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 3980
I20260928 17:46:40.849905 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:40.850309 4060 catalog_manager.cc:1347] Generated new cluster ID: 8b23d67ebcf14b41915d71bb8329cea5
I20260928 17:46:40.850349 4060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:46:40.850538 3960 external_mini_cluster.cc:1428] Setting key d589ea03de6abd0079584b0d076f228d
I20260928 17:46:40.862602 4060 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:40.863286 4060 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 17:46:40 dist-test-slave-w3bm krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:40.866379 4060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 334e8580fc2c400c86f6f4a1c362f538: Generated new TSK 0
I20260928 17:46:40.866576 4060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:46:40.867664 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:40.851263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.3.222.62:43419)
0928 17:46:40.857129 (+ 5866us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:40.857135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:40.857156 (+ 21us) server_negotiation.cc:407] Connection header received
0928 17:46:40.857237 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:40.857241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:40.857278 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:40.857383 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:40.858784 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:40.859758 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:40.860724 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:40.861113 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:40.861204 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:40.864410 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:40.864414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:40.864414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:40.864426 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:40.865844 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:40.866271 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:40.866274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:40.866274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:40.866381 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:40.866638 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:40.866639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:40.866640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:40.866900 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:40.867041 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:40.867337 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 17:46:40.867469 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5828,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:46:40.895915 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:43419
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:40.999537 4074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:40.999717 4074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:40.999742 4074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:41.000869 4074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:41.000907 4074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:41.000932 4074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:46:41.002341 4074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:43419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:41.002554 4074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:41.002748 4074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:41.007375 4079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:41.007408 4074 server_base.cc:1034] running on GCE node
W20260928 17:46:41.007481 4082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:41.007371 4080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:41.007802 4074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:41.008031 4074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:41.009166 4074 hybrid_clock.cc:648] HybridClock initialized: now 1790617601009152 us; error 31 us; skew 500 ppm
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:41.011382 4074 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:46:41.011780 4074 webserver.cc:460] Webserver started at http://127.3.222.1:33741/ using document root <none> and password file <none>
I20260928 17:46:41.011978 4074 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:41.012023 4074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:41.012140 4074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:41.013087 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "fe5cc84a839b4ad6846a80ca8e5f83c0"
format_stamp: "Formatted at 2026-09-28 17:46:41 on dist-test-slave-w3bm"
server_key: "ea77303e2d8b5a0c02ffbdc4d3cf24fb"
server_key_iv: "6ac03ae2d213d216c81409e70133d4c8"
server_key_version: "encryptionkey@0"
I20260928 17:46:41.013396 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "fe5cc84a839b4ad6846a80ca8e5f83c0"
format_stamp: "Formatted at 2026-09-28 17:46:41 on dist-test-slave-w3bm"
server_key: "ea77303e2d8b5a0c02ffbdc4d3cf24fb"
server_key_iv: "6ac03ae2d213d216c81409e70133d4c8"
server_key_version: "encryptionkey@0"
I20260928 17:46:41.014565 4074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:46:41.015282 4089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:41.015417 4074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:41.015492 4074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "fe5cc84a839b4ad6846a80ca8e5f83c0"
format_stamp: "Formatted at 2026-09-28 17:46:41 on dist-test-slave-w3bm"
server_key: "ea77303e2d8b5a0c02ffbdc4d3cf24fb"
server_key_iv: "6ac03ae2d213d216c81409e70133d4c8"
server_key_version: "encryptionkey@0"
I20260928 17:46:41.015558 4074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:41.024348 4074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:41.025928 4074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:41.026050 4074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:41.026230 4074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:41.026484 4074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:41.026535 4074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:41.026572 4074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:41.026597 4074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:41.031873 4074 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:46579
I20260928 17:46:41.031908 4202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:46579 every 8 connection(s)
I20260928 17:46:41.032171 4074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:41.040163 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:41.032774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:36421 (local address 127.3.222.62:43419)
0928 17:46:41.032847 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:41.032849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:41.032990 (+ 141us) server_negotiation.cc:407] Connection header received
0928 17:46:41.033415 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:41.033416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:41.033426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:41.033474 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:41.034303 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.034875 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.035848 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.036046 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.036100 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:41.037864 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:41.037866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:41.037866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:41.037872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:41.039056 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.039440 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.039441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.039441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.039498 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.039685 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.039686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.039686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.039804 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:41.039850 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:41.040027 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 17:46:41.040100 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:46:41.040297 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:41.032751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:43419 (local address 127.3.222.1:36421)
0928 17:46:41.032888 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:41.032920 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:41.033341 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:41.033522 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:41.033525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:41.034032 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:41.034205 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:41.034209 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.034948 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.034949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:41.035784 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:41.035787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.036102 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.036103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:41.036187 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:41.036558 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:41.036565 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:41.037790 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:41.039117 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:41.039121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:41.039123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:41.039358 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:41.039556 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:41.039556 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:41.039557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:41.039643 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:41.039844 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:41.039846 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:41.039945 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 17:46:41.040016 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 17:46:41.040123 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260928 17:46:41.040449 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 4074
I20260928 17:46:41.040534 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:46:41.040683 3960 external_mini_cluster.cc:1428] Setting key c05d1a1407a1702628d597eef9e50ed1
I20260928 17:46:41.040803 4203 heartbeater.cc:347] Connected to a master server at 127.3.222.62:43419
I20260928 17:46:41.040884 4203 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:41.041057 4203 heartbeater.cc:511] Master 127.3.222.62:43419 requested a full tablet report, sending...
I20260928 17:46:41.041510 4012 ts_manager.cc:194] Registered new tserver with Master: fe5cc84a839b4ad6846a80ca8e5f83c0 (127.3.222.1:46579)
I20260928 17:46:41.042083 4012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:36421
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:46:41.066566 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:43419
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:41.144819 4208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:41.145016 4208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:41.145038 4208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:41.146181 4208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:41.146221 4208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:41.146246 4208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:46:41.147768 4208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:43419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:41.147996 4208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:41.148208 4208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:46:41.153146 4208 server_base.cc:1034] running on GCE node
W20260928 17:46:41.153232 4213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:41.153242 4216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:41.153142 4214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:41.153587 4208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:41.153815 4208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:41.154949 4208 hybrid_clock.cc:648] HybridClock initialized: now 1790617601154930 us; error 31 us; skew 500 ppm
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:41.157297 4208 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:46:41.157729 4208 webserver.cc:460] Webserver started at http://127.3.222.2:45891/ using document root <none> and password file <none>
I20260928 17:46:41.157932 4208 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:41.157979 4208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:41.158097 4208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:41.159081 4208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "402d11077b4d4684956602accf79a7c7"
format_stamp: "Formatted at 2026-09-28 17:46:41 on dist-test-slave-w3bm"
server_key: "7641aec834d95c58d91d4f425c32642f"
server_key_iv: "0920fcecdd777c2f44051f96fca264ac"
server_key_version: "encryptionkey@0"
I20260928 17:46:41.159415 4208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "402d11077b4d4684956602accf79a7c7"
format_stamp: "Formatted at 2026-09-28 17:46:41 on dist-test-slave-w3bm"
server_key: "7641aec834d95c58d91d4f425c32642f"
server_key_iv: "0920fcecdd777c2f44051f96fca264ac"
server_key_version: "encryptionkey@0"
I20260928 17:46:41.160583 4208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:41.161262 4223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:41.161405 4208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:41.161490 4208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "402d11077b4d4684956602accf79a7c7"
format_stamp: "Formatted at 2026-09-28 17:46:41 on dist-test-slave-w3bm"
server_key: "7641aec834d95c58d91d4f425c32642f"
server_key_iv: "0920fcecdd777c2f44051f96fca264ac"
server_key_version: "encryptionkey@0"
I20260928 17:46:41.161552 4208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:41.167600 4208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:41.169221 4208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:41.169340 4208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:41.169521 4208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:41.169778 4208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:41.169809 4208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:41.169835 4208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:41.169859 4208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:41.175675 4208 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:35649
I20260928 17:46:41.175726 4336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:35649 every 8 connection(s)
I20260928 17:46:41.175997 4208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:46:41.180612 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 4208
I20260928 17:46:41.180703 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:46:41.180868 3960 external_mini_cluster.cc:1428] Setting key 5c6b84e21ef37672f337656876184e05
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:41.185140 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:41.176591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:43681 (local address 127.3.222.62:43419)
0928 17:46:41.176664 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:41.176666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:41.177207 (+ 541us) server_negotiation.cc:407] Connection header received
0928 17:46:41.177678 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:41.177679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:41.177691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:41.177743 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:41.178630 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.179131 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.180140 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.180330 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.180381 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:41.182693 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:41.182694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:41.182695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:41.182702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:41.183941 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.184293 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.184294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.184294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.184348 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.184552 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.184553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.184553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.184667 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:41.184721 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:41.184967 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 17:46:41.185051 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 17:46:41.185302 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:41.176569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:43419 (local address 127.3.222.2:43681)
0928 17:46:41.176703 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:41.176737 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:41.177587 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:41.177808 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:41.177811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:41.178346 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:41.178534 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:41.178539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.179252 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.179253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:41.180045 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:41.180048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.180393 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.180394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:41.180533 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:41.181137 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:41.181145 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:41.182602 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:41.184012 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:41.184018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:41.184020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:41.184230 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:41.184400 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:41.184401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:41.184401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:41.184500 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:41.184717 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:41.184730 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:41.184857 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 17:46:41.184940 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 17:46:41.185083 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260928 17:46:41.185753 4337 heartbeater.cc:347] Connected to a master server at 127.3.222.62:43419
I20260928 17:46:41.185854 4337 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:41.186044 4337 heartbeater.cc:511] Master 127.3.222.62:43419 requested a full tablet report, sending...
I20260928 17:46:41.186439 4012 ts_manager.cc:194] Registered new tserver with Master: 402d11077b4d4684956602accf79a7c7 (127.3.222.2:35649)
I20260928 17:46:41.186919 4012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:43681
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:46:41.206683 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:43419
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:41.286067 4342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:41.286264 4342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:41.286286 4342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:41.287436 4342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:41.287475 4342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:41.287500 4342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:46:41.288894 4342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:43419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:41.289085 4342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:41.289255 4342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:41.294234 4347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:41.294317 4348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:41.294356 4350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:41.294380 4342 server_base.cc:1034] running on GCE node
I20260928 17:46:41.294672 4342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:41.294898 4342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:41.296034 4342 hybrid_clock.cc:648] HybridClock initialized: now 1790617601296019 us; error 30 us; skew 500 ppm
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:41.298219 4342 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:46:41.298617 4342 webserver.cc:460] Webserver started at http://127.3.222.3:46233/ using document root <none> and password file <none>
I20260928 17:46:41.298820 4342 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:41.298866 4342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:41.298981 4342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:41.300004 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "d6d3265dc76c43129bf3a9708059cd00"
format_stamp: "Formatted at 2026-09-28 17:46:41 on dist-test-slave-w3bm"
server_key: "a467edb63c6633089a741c8010878ee3"
server_key_iv: "a14b89d0be957d962e6917719c5806a9"
server_key_version: "encryptionkey@0"
I20260928 17:46:41.300325 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "d6d3265dc76c43129bf3a9708059cd00"
format_stamp: "Formatted at 2026-09-28 17:46:41 on dist-test-slave-w3bm"
server_key: "a467edb63c6633089a741c8010878ee3"
server_key_iv: "a14b89d0be957d962e6917719c5806a9"
server_key_version: "encryptionkey@0"
I20260928 17:46:41.301546 4342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:46:41.302202 4357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:41.302336 4342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:46:41.302414 4342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "d6d3265dc76c43129bf3a9708059cd00"
format_stamp: "Formatted at 2026-09-28 17:46:41 on dist-test-slave-w3bm"
server_key: "a467edb63c6633089a741c8010878ee3"
server_key_iv: "a14b89d0be957d962e6917719c5806a9"
server_key_version: "encryptionkey@0"
I20260928 17:46:41.302480 4342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:41.325752 4342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:41.327540 4342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:41.327678 4342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:41.327879 4342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:41.328168 4342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:41.328200 4342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:41.328229 4342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:41.328253 4342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:41.333925 4342 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:34271
I20260928 17:46:41.333976 4470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:34271 every 8 connection(s)
I20260928 17:46:41.334241 4342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:41.341027 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 4342
I20260928 17:46:41.341104 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:46:41.341264 3960 external_mini_cluster.cc:1428] Setting key 8e4dc79c164c1922b05e36aa3aada4c9
I20260928 17:46:41.342343 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:41.334836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:53239 (local address 127.3.222.62:43419)
0928 17:46:41.334884 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:41.334886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:41.335039 (+ 153us) server_negotiation.cc:407] Connection header received
0928 17:46:41.335521 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:41.335522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:41.335531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:41.335582 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:41.336391 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.336943 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.337853 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.338024 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.338065 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:41.339897 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:41.339899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:41.339900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:41.339905 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:41.341074 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.341412 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.341413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.341413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.341472 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.341729 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.341730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.341730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.341852 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:41.341912 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:41.342183 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 17:46:41.342263 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:46:41.342514 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:41.334801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:43419 (local address 127.3.222.3:53239)
0928 17:46:41.334930 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:41.334968 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:41.335439 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:41.335634 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:41.335637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:41.336149 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:41.336327 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:41.336332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.337011 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.337012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:41.337791 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:41.337793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.338074 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.338075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:41.338161 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:41.338522 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:41.338529 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:41.339821 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:41.341146 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:41.341152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:41.341154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:41.341350 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:41.341539 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:41.341540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:41.341541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:41.341670 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:41.341910 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:41.341915 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:41.342047 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 17:46:41.342164 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 17:46:41.342296 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260928 17:46:41.342926 4471 heartbeater.cc:347] Connected to a master server at 127.3.222.62:43419
I20260928 17:46:41.343004 4471 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:41.343127 4471 heartbeater.cc:511] Master 127.3.222.62:43419 requested a full tablet report, sending...
I20260928 17:46:41.343438 4012 ts_manager.cc:194] Registered new tserver with Master: d6d3265dc76c43129bf3a9708059cd00 (127.3.222.3:34271)
I20260928 17:46:41.343780 4012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:53239
I20260928 17:46:41.352996 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:46:41.374037 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:41.366884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.3.222.1:46579)
0928 17:46:41.367021 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:41.367025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:41.367036 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:46:41.367062 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:41.367063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:41.367077 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:41.367137 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:41.367656 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.368353 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.369048 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.369370 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.369421 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:41.371490 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:41.371500 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:41.371501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:41.371509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:41.373022 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.373315 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.373319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.373319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.373374 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.373529 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.373530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.373530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.373658 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:41.373820 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:41.373861 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 17:46:41.373932 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
W20260928 17:46:41.374343 4117 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52342
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:46:41.394737 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:41.388396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.3.222.1:46579)
0928 17:46:41.388454 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:41.388456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:41.388469 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:46:41.388532 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:41.388532 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:41.388543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:41.388594 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:41.389094 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.389565 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.390243 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:41.390442 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:41.390491 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:41.392556 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:41.392558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:41.392558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:41.392564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:41.393798 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.394086 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.394087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.394087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.394136 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:41.394292 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:41.394293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:41.394293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:41.394383 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:41.394485 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:41.394565 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 17:46:41.394646 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:46:41.395241 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 4074
I20260928 17:46:41.399269 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 4208
I20260928 17:46:41.403842 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 4342
I20260928 17:46:41.407291 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 3980
2026-09-28T17:46:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1760 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[4485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[4485](info): set up 2 sockets
Sep 28 17:46:41 dist-test-slave-w3bm krb5kdc[4485](info): commencing operation
krb5kdc: starting...
W20260928 17:46:42.362413 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:46:42 dist-test-slave-w3bm krb5kdc[4485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:46:42.474010 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:39441
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:39441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:42.551095 4498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:42.551316 4498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:42.551345 4498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:42.552462 4498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:42.552502 4498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:42.552520 4498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:42.552536 4498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:42.552551 4498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:42.554046 4498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:39441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:39441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:42.554275 4498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:42.554474 4498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:42.558926 4506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:42.559005 4498 server_base.cc:1034] running on GCE node
W20260928 17:46:42.559075 4503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:42.558955 4504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:42.559475 4498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:42.559795 4498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:42.560957 4498 hybrid_clock.cc:648] HybridClock initialized: now 1790617602560939 us; error 44 us; skew 500 ppm
Sep 28 17:46:42 dist-test-slave-w3bm krb5kdc[4485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:42.563158 4498 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:46:42.563547 4498 webserver.cc:460] Webserver started at http://127.3.222.62:44311/ using document root <none> and password file <none>
I20260928 17:46:42.563724 4498 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:42.563761 4498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:42.563844 4498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:42.564810 4498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "4b987293f3634d88ace7aea22be15c63"
format_stamp: "Formatted at 2026-09-28 17:46:42 on dist-test-slave-w3bm"
server_key: "5a415b6db4fc07106f56bd278fb18ae2"
server_key_iv: "ecbd3c0173c4f7e36ea4d1e9d4e6a86c"
server_key_version: "encryptionkey@0"
I20260928 17:46:42.565099 4498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "4b987293f3634d88ace7aea22be15c63"
format_stamp: "Formatted at 2026-09-28 17:46:42 on dist-test-slave-w3bm"
server_key: "5a415b6db4fc07106f56bd278fb18ae2"
server_key_iv: "ecbd3c0173c4f7e36ea4d1e9d4e6a86c"
server_key_version: "encryptionkey@0"
I20260928 17:46:42.566224 4498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:42.566823 4513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:42.566998 4498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:42.567075 4498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "4b987293f3634d88ace7aea22be15c63"
format_stamp: "Formatted at 2026-09-28 17:46:42 on dist-test-slave-w3bm"
server_key: "5a415b6db4fc07106f56bd278fb18ae2"
server_key_iv: "ecbd3c0173c4f7e36ea4d1e9d4e6a86c"
server_key_version: "encryptionkey@0"
I20260928 17:46:42.567126 4498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:42.584431 4498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:42.586159 4498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:42.586263 4498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:42.589926 4498 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:39441
I20260928 17:46:42.589977 4565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:39441 every 8 connection(s)
I20260928 17:46:42.590271 4498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:42.591073 4566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:42.593487 4566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63: Bootstrap starting.
I20260928 17:46:42.594045 4566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:42.594291 4566 log.cc:824] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:42.594957 4566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63: No bootstrap required, opened a new log
I20260928 17:46:42.596482 4566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b987293f3634d88ace7aea22be15c63" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39441 } }
I20260928 17:46:42.596617 4566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:42.596650 4566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b987293f3634d88ace7aea22be15c63, State: Initialized, Role: FOLLOWER
I20260928 17:46:42.596796 4566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b987293f3634d88ace7aea22be15c63" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39441 } }
I20260928 17:46:42.596881 4566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:42.596913 4566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:42.596961 4566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:42.597448 4566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b987293f3634d88ace7aea22be15c63" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39441 } }
I20260928 17:46:42.597540 4566 leader_election.cc:302] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b987293f3634d88ace7aea22be15c63; no voters:
I20260928 17:46:42.597689 4566 leader_election.cc:288] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:42.597724 4569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:42.598012 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 4498
I20260928 17:46:42.598088 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:42.598225 3960 external_mini_cluster.cc:1428] Setting key 706b71479ed62d3a457c970da59ba0c8
I20260928 17:46:42.597853 4569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [term 1 LEADER]: Becoming Leader. State: Replica: 4b987293f3634d88ace7aea22be15c63, State: Running, Role: LEADER
I20260928 17:46:42.597935 4566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:42.598479 4569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b987293f3634d88ace7aea22be15c63" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39441 } }
I20260928 17:46:42.599119 4570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b987293f3634d88ace7aea22be15c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b987293f3634d88ace7aea22be15c63" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39441 } } }
I20260928 17:46:42.599115 4571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b987293f3634d88ace7aea22be15c63. Latest consensus state: current_term: 1 leader_uuid: "4b987293f3634d88ace7aea22be15c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b987293f3634d88ace7aea22be15c63" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39441 } } }
I20260928 17:46:42.599261 4571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:42.599255 4570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:42.599502 4575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:42.600020 4575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:46:42.601450 4575 catalog_manager.cc:1347] Generated new cluster ID: 4c4822d196ba458e99808c4f287961de
I20260928 17:46:42.601495 4575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:46:42 dist-test-slave-w3bm krb5kdc[4485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:42.609676 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:42.598617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.3.222.62:39441)
0928 17:46:42.599065 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:42.599076 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:42.599470 (+ 394us) server_negotiation.cc:407] Connection header received
0928 17:46:42.599604 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:42.599612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:42.599677 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:42.600620 (+ 943us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:42.600724 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.601836 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.602971 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.603569 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.603659 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:42.606249 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:42.606255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:42.606256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:42.606270 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:42.608146 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:42.608497 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:42.608500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:42.608500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:42.608623 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:42.608791 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:42.608792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:42.608792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:42.609104 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:42.609254 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:42.609328 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 17:46:42.609468 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":65,"threads_started":1}
I20260928 17:46:42.611913 4575 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:42.612586 4575 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:46:42.617486 4575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b987293f3634d88ace7aea22be15c63: Generated new TSK 0
I20260928 17:46:42.617640 4575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:46:42.647096 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:39441
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:42.730661 4592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:42.730865 4592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:42.730896 4592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:42.732085 4592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:42.732125 4592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:42.732158 4592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:46:42.733549 4592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:39441
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:42.733767 4592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:42.733958 4592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:42.738520 4600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:42.738523 4597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:42.738523 4598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:42.738884 4592 server_base.cc:1034] running on GCE node
I20260928 17:46:42.739113 4592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:42.739351 4592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:42.740491 4592 hybrid_clock.cc:648] HybridClock initialized: now 1790617602740472 us; error 32 us; skew 500 ppm
Sep 28 17:46:42 dist-test-slave-w3bm krb5kdc[4485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:42.742815 4592 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:46:42.743181 4592 webserver.cc:460] Webserver started at http://127.3.222.1:41843/ using document root <none> and password file <none>
I20260928 17:46:42.743402 4592 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:42.743449 4592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:42.743563 4592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:42.744517 4592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "5b7309721b5f42fc8485b285a05bf90c"
format_stamp: "Formatted at 2026-09-28 17:46:42 on dist-test-slave-w3bm"
server_key: "012c51d13d9d50f6da17bb3af669becc"
server_key_iv: "e6bf8e27f029a084f82cb1d53abff2c7"
server_key_version: "encryptionkey@0"
I20260928 17:46:42.744820 4592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "5b7309721b5f42fc8485b285a05bf90c"
format_stamp: "Formatted at 2026-09-28 17:46:42 on dist-test-slave-w3bm"
server_key: "012c51d13d9d50f6da17bb3af669becc"
server_key_iv: "e6bf8e27f029a084f82cb1d53abff2c7"
server_key_version: "encryptionkey@0"
I20260928 17:46:42.745940 4592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:46:42.746527 4607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:42.746668 4592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:46:42.746747 4592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "5b7309721b5f42fc8485b285a05bf90c"
format_stamp: "Formatted at 2026-09-28 17:46:42 on dist-test-slave-w3bm"
server_key: "012c51d13d9d50f6da17bb3af669becc"
server_key_iv: "e6bf8e27f029a084f82cb1d53abff2c7"
server_key_version: "encryptionkey@0"
I20260928 17:46:42.746805 4592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:42.764540 4592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:42.766292 4592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:42.766417 4592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:42.766609 4592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:42.766892 4592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:42.766923 4592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:42.766952 4592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:42.766974 4592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:42.772575 4592 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:40461
I20260928 17:46:42.772608 4720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:40461 every 8 connection(s)
I20260928 17:46:42.772925 4592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:46:42 dist-test-slave-w3bm krb5kdc[4485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:42.781335 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:42.773519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:38207 (local address 127.3.222.62:39441)
0928 17:46:42.773567 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:42.773569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:42.773751 (+ 182us) server_negotiation.cc:407] Connection header received
0928 17:46:42.774665 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:42.774666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:42.774681 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:42.774720 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:42.775320 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.775811 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.776854 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.777053 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.777115 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:42.779053 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:42.779055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:42.779055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:42.779063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:42.780153 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:42.780506 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:42.780507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:42.780507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:42.780558 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:42.780757 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:42.780758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:42.780758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:42.780869 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:42.780922 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:42.781168 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 17:46:42.781249 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:46:42.781500 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:42.773500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:39441 (local address 127.3.222.1:38207)
0928 17:46:42.773642 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:42.773683 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:42.774568 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:42.774783 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:42.774786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:42.775027 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:42.775225 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:42.775230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.775899 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.775899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:42.776715 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:42.776718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.777121 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.777122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:42.777216 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:42.777645 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:42.777653 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:42.778965 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:42.780224 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:42.780230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:42.780232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:42.780443 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:42.780610 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:42.780611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:42.780611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:42.780704 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:42.780922 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:42.780928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:42.781053 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 17:46:42.781137 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:46:42.781278 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260928 17:46:42.781842 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 4592
I20260928 17:46:42.781916 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:46:42.781944 4721 heartbeater.cc:347] Connected to a master server at 127.3.222.62:39441
I20260928 17:46:42.782030 4721 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:42.782074 3960 external_mini_cluster.cc:1428] Setting key 2b067bfb17b77adcf03d9110dc4394e6
I20260928 17:46:42.782205 4721 heartbeater.cc:511] Master 127.3.222.62:39441 requested a full tablet report, sending...
I20260928 17:46:42.782733 4529 ts_manager.cc:194] Registered new tserver with Master: 5b7309721b5f42fc8485b285a05bf90c (127.3.222.1:40461)
I20260928 17:46:42.783399 4529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:38207
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:46:42.806183 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:39441
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:42.883761 4726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:42.883961 4726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:42.883983 4726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:42.885116 4726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:42.885157 4726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:42.885182 4726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:46:42.886619 4726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:39441
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:42.886842 4726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:42.887038 4726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:46:42.891778 4726 server_base.cc:1034] running on GCE node
W20260928 17:46:42.891739 4732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:42.891770 4731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:42.891734 4734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:42.892148 4726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:42.892340 4726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:42.893499 4726 hybrid_clock.cc:648] HybridClock initialized: now 1790617602893487 us; error 25 us; skew 500 ppm
Sep 28 17:46:42 dist-test-slave-w3bm krb5kdc[4485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:42.895938 4726 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:46:42.896320 4726 webserver.cc:460] Webserver started at http://127.3.222.2:44913/ using document root <none> and password file <none>
I20260928 17:46:42.896502 4726 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:42.896541 4726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:42.896657 4726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:42.897627 4726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "77ad033427324031ae48471216bdcc65"
format_stamp: "Formatted at 2026-09-28 17:46:42 on dist-test-slave-w3bm"
server_key: "be4ba85ceb19b50120dba4c516082a3f"
server_key_iv: "a21dbdb80f521a2ce3926ecd32db18ab"
server_key_version: "encryptionkey@0"
I20260928 17:46:42.897967 4726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "77ad033427324031ae48471216bdcc65"
format_stamp: "Formatted at 2026-09-28 17:46:42 on dist-test-slave-w3bm"
server_key: "be4ba85ceb19b50120dba4c516082a3f"
server_key_iv: "a21dbdb80f521a2ce3926ecd32db18ab"
server_key_version: "encryptionkey@0"
I20260928 17:46:42.899277 4726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:42.899976 4741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:42.900110 4726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:42.900192 4726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "77ad033427324031ae48471216bdcc65"
format_stamp: "Formatted at 2026-09-28 17:46:42 on dist-test-slave-w3bm"
server_key: "be4ba85ceb19b50120dba4c516082a3f"
server_key_iv: "a21dbdb80f521a2ce3926ecd32db18ab"
server_key_version: "encryptionkey@0"
I20260928 17:46:42.900256 4726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:42.913311 4726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:42.914851 4726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:42.914970 4726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:42.915153 4726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:42.915419 4726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:42.915450 4726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:42.915477 4726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:42.915493 4726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:42.920616 4726 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:45567
I20260928 17:46:42.920758 4854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:45567 every 8 connection(s)
I20260928 17:46:42.920997 4726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:46:42 dist-test-slave-w3bm krb5kdc[4485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:42.930373 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 4726
I20260928 17:46:42.930461 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:46:42.930619 3960 external_mini_cluster.cc:1428] Setting key 94618276c1339f2b0af18eef3c220015
I20260928 17:46:42.931576 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:42.921934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:37609 (local address 127.3.222.62:39441)
0928 17:46:42.922003 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:42.922005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:42.922016 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:46:42.923180 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:42.923183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:42.923196 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:42.923278 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:42.924261 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.925010 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.926364 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.926669 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.926755 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:42.928954 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:42.928956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:42.928956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:42.928963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:42.930138 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:42.930593 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:42.930594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:42.930595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:42.930653 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:42.930936 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:42.930937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:42.930938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:42.931056 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:42.931112 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:42.931407 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 17:46:42.931490 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:46:42.931734 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:42.921661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:39441 (local address 127.3.222.2:37609)
0928 17:46:42.921864 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:42.921925 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:42.923063 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:42.923356 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:42.923362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:42.923697 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:42.923995 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:42.924002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.925114 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.925115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:42.926279 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:42.926282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:42.926752 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:42.926753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:42.926884 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:42.927370 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:42.927381 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:42.928872 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:42.930231 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:42.930238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:42.930240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:42.930524 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:42.930726 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:42.930727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:42.930727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:42.930883 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:42.931110 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:42.931116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:42.931287 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 17:46:42.931378 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 17:46:42.931523 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260928 17:46:42.932125 4855 heartbeater.cc:347] Connected to a master server at 127.3.222.62:39441
I20260928 17:46:42.932220 4855 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:42.932397 4855 heartbeater.cc:511] Master 127.3.222.62:39441 requested a full tablet report, sending...
I20260928 17:46:42.932767 4529 ts_manager.cc:194] Registered new tserver with Master: 77ad033427324031ae48471216bdcc65 (127.3.222.2:45567)
I20260928 17:46:42.933108 4529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:37609
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:46:42.953420 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:39441
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:43.030318 4860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:43.030501 4860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:43.030522 4860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:43.031651 4860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:43.031687 4860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:43.031711 4860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:46:43.033128 4860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:39441
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:43.033330 4860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:43.033526 4860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:43.038050 4865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:43.038085 4866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:43.038169 4860 server_base.cc:1034] running on GCE node
W20260928 17:46:43.038048 4868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:43.038429 4860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:43.038645 4860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:43.039767 4860 hybrid_clock.cc:648] HybridClock initialized: now 1790617603039759 us; error 26 us; skew 500 ppm
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[4485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:43.042011 4860 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:46:43.042378 4860 webserver.cc:460] Webserver started at http://127.3.222.3:40795/ using document root <none> and password file <none>
I20260928 17:46:43.042573 4860 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:43.042615 4860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:43.042721 4860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:43.043735 4860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "38a6e850f8be4e8892aef785432b07a5"
format_stamp: "Formatted at 2026-09-28 17:46:43 on dist-test-slave-w3bm"
server_key: "703ce1dd3570265c92eb8e3224da8388"
server_key_iv: "0ec6cdea19e306dc0d445d477fbe7b75"
server_key_version: "encryptionkey@0"
I20260928 17:46:43.044049 4860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "38a6e850f8be4e8892aef785432b07a5"
format_stamp: "Formatted at 2026-09-28 17:46:43 on dist-test-slave-w3bm"
server_key: "703ce1dd3570265c92eb8e3224da8388"
server_key_iv: "0ec6cdea19e306dc0d445d477fbe7b75"
server_key_version: "encryptionkey@0"
I20260928 17:46:43.045193 4860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:43.045884 4875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:43.046037 4860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:43.046115 4860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "38a6e850f8be4e8892aef785432b07a5"
format_stamp: "Formatted at 2026-09-28 17:46:43 on dist-test-slave-w3bm"
server_key: "703ce1dd3570265c92eb8e3224da8388"
server_key_iv: "0ec6cdea19e306dc0d445d477fbe7b75"
server_key_version: "encryptionkey@0"
I20260928 17:46:43.046171 4860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:43.059983 4860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:43.061864 4860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:43.062005 4860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:43.062203 4860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:43.062484 4860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:43.062525 4860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:43.062553 4860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:43.062572 4860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:43.067790 4860 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:46071
I20260928 17:46:43.067845 4988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:46071 every 8 connection(s)
I20260928 17:46:43.068121 4860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[4485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:43.077375 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.068773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:54853 (local address 127.3.222.62:39441)
0928 17:46:43.068817 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.068818 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.069010 (+ 192us) server_negotiation.cc:407] Connection header received
0928 17:46:43.070002 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.070003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.070015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.070056 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:43.070683 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.071174 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.072172 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.072361 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.072415 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.074495 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:43.074497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:43.074497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:43.074504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:43.076093 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.076485 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.076487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.076487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.076548 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.076771 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.076772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.076772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.076921 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:43.076981 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.077231 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.077312 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:46:43.077517 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 4860
I20260928 17:46:43.077546 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.068746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:39441 (local address 127.3.222.3:54853)
0928 17:46:43.068899 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:43.068933 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:43.069905 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:43.070123 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:43.070127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:43.070406 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:43.070596 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.070601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.071274 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.071275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.072094 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.072096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.072425 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.072426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.072558 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.073043 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:43.073051 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:43.074388 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:43.076173 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:43.076179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:43.076180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:43.076390 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:43.076605 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:43.076605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:43.076605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:43.076700 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:43.076975 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:43.076978 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:43.077090 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 17:46:43.077167 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:46:43.077294 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260928 17:46:43.077610 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:46:43.077770 3960 external_mini_cluster.cc:1428] Setting key 5a16cbf71f5a0c76b8c1a4180ef0a9a2
I20260928 17:46:43.078049 4989 heartbeater.cc:347] Connected to a master server at 127.3.222.62:39441
I20260928 17:46:43.078152 4989 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:43.078428 4989 heartbeater.cc:511] Master 127.3.222.62:39441 requested a full tablet report, sending...
I20260928 17:46:43.078796 4529 ts_manager.cc:194] Registered new tserver with Master: 38a6e850f8be4e8892aef785432b07a5 (127.3.222.3:46071)
I20260928 17:46:43.079154 4529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:54853
I20260928 17:46:43.089434 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:46:43.097339 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.091008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.3.222.62:39441)
0928 17:46:43.091056 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.091058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.091161 (+ 103us) server_negotiation.cc:407] Connection header received
0928 17:46:43.091237 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.091238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.091248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.091294 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:43.091823 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.092297 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.093002 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.093184 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.093236 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.094321 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:43.094323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:43.094323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:43.094330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:43.096182 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.096602 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.096604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.096604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.096658 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.096819 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.096820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.096820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.096975 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:43.097066 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.097176 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.097250 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:46:43.099566 4529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:46:43.101208 4529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:43.107074 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.103250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.3.222.3:46071)
0928 17:46:43.103575 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.103580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.103590 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:46:43.103632 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.103634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.103650 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.103728 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:43.103909 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.104769 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.106175 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.106689 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.106750 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.106822 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.106959 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.106997 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":74,"threads_started":1}
I20260928 17:46:43.107117 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.103349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:45567 (local address 127.0.0.1:49138)
0928 17:46:43.103517 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:43.103528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:43.103569 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:43.103869 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:43.103869 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:43.103871 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:43.103922 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.103924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.105172 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.105173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.105992 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.105996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.106867 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.106868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.106930 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.106935 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:46:43.106964 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 17:46:43.107005 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":40,"thread_start_us":42,"threads_started":1}
I20260928 17:46:43.107112 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.103138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:46071 (local address 127.0.0.1:39006)
0928 17:46:43.103422 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:43.103433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:43.103482 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:43.103765 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:43.103768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:43.103770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:43.103844 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.103846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.104856 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.104857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.106084 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.106087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.106775 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.106776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.106865 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.106879 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:46:43.106938 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 17:46:43.106992 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
I20260928 17:46:43.107375 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.103132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:40461 (local address 127.0.0.1:53964)
0928 17:46:43.103273 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:43.103314 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:43.103379 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:43.103918 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:43.103919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:43.103921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:43.103953 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.103954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.105161 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.105162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.106221 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.106223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.107127 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.107128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.107182 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.107185 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:46:43.107281 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 17:46:43.107315 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260928 17:46:43.107383 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.103544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.3.222.2:45567)
0928 17:46:43.103678 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.103683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.103692 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:46:43.103766 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.103768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.103782 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.103856 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:43.103982 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.105055 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.106073 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.106771 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.107153 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.107243 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.107269 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.107308 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260928 17:46:43.107600 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.103345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.3.222.1:40461)
0928 17:46:43.103598 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.103602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.103610 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:46:43.103641 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.103643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.103656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.103728 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:43.104023 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.105072 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.106290 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.106991 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.107044 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.107096 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.107507 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.107545 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
I20260928 17:46:43.108078 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 1d54591da22b4213ae1aeb3054d45521 (DEFAULT_TABLE table=test-table [id=c98beb7a397a4e8ea50c4d026ff746c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:43.108120 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 1d54591da22b4213ae1aeb3054d45521 (DEFAULT_TABLE table=test-table [id=c98beb7a397a4e8ea50c4d026ff746c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:43.108402 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d54591da22b4213ae1aeb3054d45521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:43.108426 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d54591da22b4213ae1aeb3054d45521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:43.110769 5003 tablet_bootstrap.cc:492] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65: Bootstrap starting.
I20260928 17:46:43.110780 5004 tablet_bootstrap.cc:492] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c: Bootstrap starting.
I20260928 17:46:43.111485 5004 tablet_bootstrap.cc:654] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:43.111593 5003 tablet_bootstrap.cc:654] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:43.111570 4923 tablet_service.cc:1467] Processing CreateTablet for tablet 1d54591da22b4213ae1aeb3054d45521 (DEFAULT_TABLE table=test-table [id=c98beb7a397a4e8ea50c4d026ff746c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:43.111897 4923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d54591da22b4213ae1aeb3054d45521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:43.111955 5004 log.cc:824] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:43.112088 5003 log.cc:824] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:43.112659 5004 tablet_bootstrap.cc:492] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c: No bootstrap required, opened a new log
I20260928 17:46:43.112687 5003 tablet_bootstrap.cc:492] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65: No bootstrap required, opened a new log
I20260928 17:46:43.112731 5004 ts_tablet_manager.cc:1397] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:46:43.112738 5003 ts_tablet_manager.cc:1397] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:43.114442 5003 raft_consensus.cc:348] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } }
I20260928 17:46:43.114617 5003 raft_consensus.cc:374] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:43.114655 5003 raft_consensus.cc:729] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ad033427324031ae48471216bdcc65, State: Initialized, Role: FOLLOWER
I20260928 17:46:43.114600 5004 raft_consensus.cc:348] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } }
I20260928 17:46:43.114763 5004 raft_consensus.cc:374] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:43.114794 5004 raft_consensus.cc:729] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7309721b5f42fc8485b285a05bf90c, State: Initialized, Role: FOLLOWER
I20260928 17:46:43.114796 5003 consensus_queue.cc:260] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } }
I20260928 17:46:43.114895 5004 consensus_queue.cc:260] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } }
I20260928 17:46:43.115063 5003 ts_tablet_manager.cc:1428] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:46:43.115141 5004 ts_tablet_manager.cc:1428] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 17:46:43.115183 4855 heartbeater.cc:503] Master 127.3.222.62:39441 was elected leader, sending a full tablet report...
I20260928 17:46:43.115273 4721 heartbeater.cc:503] Master 127.3.222.62:39441 was elected leader, sending a full tablet report...
I20260928 17:46:43.115404 5007 tablet_bootstrap.cc:492] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5: Bootstrap starting.
I20260928 17:46:43.115948 5007 tablet_bootstrap.cc:654] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:43.116456 5007 log.cc:824] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:43.117066 5007 tablet_bootstrap.cc:492] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5: No bootstrap required, opened a new log
I20260928 17:46:43.117116 5007 ts_tablet_manager.cc:1397] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:43.118348 5007 raft_consensus.cc:348] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } }
I20260928 17:46:43.118472 5007 raft_consensus.cc:374] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:43.118492 5007 raft_consensus.cc:729] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a6e850f8be4e8892aef785432b07a5, State: Initialized, Role: FOLLOWER
I20260928 17:46:43.118579 5007 consensus_queue.cc:260] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } }
I20260928 17:46:43.118832 5007 ts_tablet_manager.cc:1428] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:46:43.118854 4989 heartbeater.cc:503] Master 127.3.222.62:39441 was elected leader, sending a full tablet report...
W20260928 17:46:43.172075 4856 tablet.cc:2369] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:46:43.273540 4722 tablet.cc:2369] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:46:43.318608 4990 tablet.cc:2369] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:46:43.358619 5008 raft_consensus.cc:482] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:43.358744 5008 raft_consensus.cc:504] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } }
I20260928 17:46:43.359112 5008 leader_election.cc:288] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7309721b5f42fc8485b285a05bf90c (127.3.222.1:40461), 38a6e850f8be4e8892aef785432b07a5 (127.3.222.3:46071)
I20260928 17:46:43.362936 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.359371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:55603 (local address 127.3.222.3:46071)
0928 17:46:43.359418 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.359420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.359431 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:46:43.359538 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.359540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.359555 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.359634 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:43.359803 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.360548 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.362070 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.362634 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.362694 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.362703 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.362840 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.362874 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:46:43.362962 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.359360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:47115 (local address 127.3.222.1:40461)
0928 17:46:43.359410 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.359412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.359719 (+ 307us) server_negotiation.cc:407] Connection header received
0928 17:46:43.359773 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.359774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.359825 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.359880 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:43.359992 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.360673 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.362042 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.362593 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.362676 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.362688 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.362831 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.362869 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:46:43.362942 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.359307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:40461 (local address 127.3.222.2:47115)
0928 17:46:43.359650 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:43.359660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:43.359719 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:43.359888 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:43.359889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:43.359890 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:43.359933 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.359935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.360755 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.360755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.361937 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.361940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.362681 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.362682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.362752 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.362757 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:46:43.362809 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 17:46:43.362849 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":113,"thread_start_us":194,"threads_started":1}
I20260928 17:46:43.362951 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.359307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:46071 (local address 127.3.222.2:55603)
0928 17:46:43.359358 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:43.359369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:43.359419 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:43.359655 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:43.359656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:43.359659 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:43.359735 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.359737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.360667 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.360668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.361980 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:43.361982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.362706 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.362706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:43.362779 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.362781 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:46:43.362830 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 17:46:43.362858 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":33}
I20260928 17:46:43.363327 4943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d54591da22b4213ae1aeb3054d45521" candidate_uuid: "77ad033427324031ae48471216bdcc65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a6e850f8be4e8892aef785432b07a5" is_pre_election: true
I20260928 17:46:43.363472 4943 raft_consensus.cc:2454] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ad033427324031ae48471216bdcc65 in term 0.
I20260928 17:46:43.363472 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d54591da22b4213ae1aeb3054d45521" candidate_uuid: "77ad033427324031ae48471216bdcc65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7309721b5f42fc8485b285a05bf90c" is_pre_election: true
I20260928 17:46:43.363619 4675 raft_consensus.cc:2454] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ad033427324031ae48471216bdcc65 in term 0.
I20260928 17:46:43.363685 4745 leader_election.cc:302] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a6e850f8be4e8892aef785432b07a5, 77ad033427324031ae48471216bdcc65; no voters:
I20260928 17:46:43.363816 5008 raft_consensus.cc:2781] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:43.363868 5008 raft_consensus.cc:482] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:43.363895 5008 raft_consensus.cc:3037] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:43.364545 5008 raft_consensus.cc:504] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } }
I20260928 17:46:43.364653 5008 leader_election.cc:288] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [CANDIDATE]: Term 1 election: Requested vote from peers 5b7309721b5f42fc8485b285a05bf90c (127.3.222.1:40461), 38a6e850f8be4e8892aef785432b07a5 (127.3.222.3:46071)
I20260928 17:46:43.364805 4943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d54591da22b4213ae1aeb3054d45521" candidate_uuid: "77ad033427324031ae48471216bdcc65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a6e850f8be4e8892aef785432b07a5"
I20260928 17:46:43.364823 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d54591da22b4213ae1aeb3054d45521" candidate_uuid: "77ad033427324031ae48471216bdcc65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7309721b5f42fc8485b285a05bf90c"
I20260928 17:46:43.364879 4675 raft_consensus.cc:3037] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:43.364879 4943 raft_consensus.cc:3037] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:43.365677 4675 raft_consensus.cc:2454] T 1d54591da22b4213ae1aeb3054d45521 P 5b7309721b5f42fc8485b285a05bf90c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ad033427324031ae48471216bdcc65 in term 1.
I20260928 17:46:43.365677 4943 raft_consensus.cc:2454] T 1d54591da22b4213ae1aeb3054d45521 P 38a6e850f8be4e8892aef785432b07a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ad033427324031ae48471216bdcc65 in term 1.
I20260928 17:46:43.365873 4744 leader_election.cc:302] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7309721b5f42fc8485b285a05bf90c, 77ad033427324031ae48471216bdcc65; no voters:
I20260928 17:46:43.365970 5008 raft_consensus.cc:2781] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:43.366097 5008 raft_consensus.cc:686] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [term 1 LEADER]: Becoming Leader. State: Replica: 77ad033427324031ae48471216bdcc65, State: Running, Role: LEADER
I20260928 17:46:43.366187 5008 consensus_queue.cc:237] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } }
I20260928 17:46:43.367051 4529 catalog_manager.cc:5495] T 1d54591da22b4213ae1aeb3054d45521 P 77ad033427324031ae48471216bdcc65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ad033427324031ae48471216bdcc65 (127.3.222.2). New cstate: current_term: 1 leader_uuid: "77ad033427324031ae48471216bdcc65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7309721b5f42fc8485b285a05bf90c" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38a6e850f8be4e8892aef785432b07a5" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ad033427324031ae48471216bdcc65" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45567 } health_report { overall_health: HEALTHY } } }
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[4485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:46:43.392045 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.385263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.3.222.1:40461)
0928 17:46:43.385319 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.385321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.385391 (+ 70us) server_negotiation.cc:407] Connection header received
0928 17:46:43.385439 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.385440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.385448 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.385494 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:43.386029 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.386524 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.387290 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.387480 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.387539 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.389583 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:43.389593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:43.389594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:43.389617 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:43.391106 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.391446 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.391450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.391450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.391517 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.391652 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.391652 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.391653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.391778 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:43.391882 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.391911 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.391972 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[4485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[4485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:46:43.413169 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.406976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.3.222.1:40461)
0928 17:46:43.407060 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.407062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.407071 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:46:43.407136 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.407137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.407147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.407194 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:43.407710 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.408183 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.408880 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.409077 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.409129 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.411106 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:43.411108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:43.411108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:43.411114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:43.412309 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.412601 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.412601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.412602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.412648 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.412776 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.412776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.412776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.412860 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:43.412926 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.413047 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.413104 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[4485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[4485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:46:43.432664 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:43.426689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53994 (local address 127.3.222.1:40461)
0928 17:46:43.426757 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:43.426759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:43.426849 (+ 90us) server_negotiation.cc:407] Connection header received
0928 17:46:43.426874 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:43.426875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:43.426885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:43.426931 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:43.427504 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.427980 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.428667 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:43.428848 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:43.428907 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:43.430848 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:43.430850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:43.430850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:43.430856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:43.431900 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.432170 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.432171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.432171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.432214 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:43.432330 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:43.432331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:43.432331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:43.432408 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:43.432476 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:43.432550 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 17:46:43.432601 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 17:46:43.433079 4635 tablet_service.cc:2908] Rejecting scan request for tablet 1d54591da22b4213ae1aeb3054d45521: Uninitialized: clean time has not yet been initialized
I20260928 17:46:43.433780 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 4592
I20260928 17:46:43.437899 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 4726
I20260928 17:46:43.441713 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 4860
I20260928 17:46:43.445204 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 4498
2026-09-28T17:46:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2038 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[5027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[5027](info): set up 2 sockets
Sep 28 17:46:43 dist-test-slave-w3bm krb5kdc[5027](info): commencing operation
krb5kdc: starting...
W20260928 17:46:44.406616 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:46:44 dist-test-slave-w3bm krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:46:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:46:44.513691 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36743
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:36743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:44.590945 5040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:44.591143 5040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:44.591162 5040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:44.592278 5040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:44.592314 5040 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 17:46:44.592326 5040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:44.592338 5040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:44.592350 5040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:44.592360 5040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:44.593724 5040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:36743
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:44.593897 5040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:44.594060 5040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:46:44.598536 5040 server_base.cc:1034] running on GCE node
W20260928 17:46:44.598510 5048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:44.598546 5045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:44.598510 5046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:44.598865 5040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:44.599120 5040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:44.600286 5040 hybrid_clock.cc:648] HybridClock initialized: now 1790617604600261 us; error 31 us; skew 500 ppm
Sep 28 17:46:44 dist-test-slave-w3bm krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:44.602550 5040 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:46:44.602929 5040 webserver.cc:460] Webserver started at http://127.3.222.62:41299/ using document root <none> and password file <none>
I20260928 17:46:44.603124 5040 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:44.603163 5040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:44.603281 5040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:44.604224 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "0b3699861e864e159108b1f174ca3c4d"
format_stamp: "Formatted at 2026-09-28 17:46:44 on dist-test-slave-w3bm"
server_key: "c128655fff7b924fe7a5c895d89c3675"
server_key_iv: "f1181b1db07d4cfa0935049562b4ce6a"
server_key_version: "encryptionkey@0"
I20260928 17:46:44.604485 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "0b3699861e864e159108b1f174ca3c4d"
format_stamp: "Formatted at 2026-09-28 17:46:44 on dist-test-slave-w3bm"
server_key: "c128655fff7b924fe7a5c895d89c3675"
server_key_iv: "f1181b1db07d4cfa0935049562b4ce6a"
server_key_version: "encryptionkey@0"
I20260928 17:46:44.605609 5040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:44.606223 5055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:44.606356 5040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:44.606427 5040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "0b3699861e864e159108b1f174ca3c4d"
format_stamp: "Formatted at 2026-09-28 17:46:44 on dist-test-slave-w3bm"
server_key: "c128655fff7b924fe7a5c895d89c3675"
server_key_iv: "f1181b1db07d4cfa0935049562b4ce6a"
server_key_version: "encryptionkey@0"
I20260928 17:46:44.606488 5040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:44.618403 5040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:44.620153 5040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:44.620265 5040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:44.624298 5040 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:36743
I20260928 17:46:44.624367 5117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:36743 every 8 connection(s)
I20260928 17:46:44.624620 5040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:44.625231 5118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:44.627401 5118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d: Bootstrap starting.
I20260928 17:46:44.627521 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 5040
I20260928 17:46:44.627604 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:44.627754 3960 external_mini_cluster.cc:1428] Setting key eb024f75d551b865cd8fe2bff2b61c5f
I20260928 17:46:44.628042 5118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:44.628345 5118 log.cc:824] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:44.628912 5118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d: No bootstrap required, opened a new log
I20260928 17:46:44.630622 5118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3699861e864e159108b1f174ca3c4d" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36743 } }
I20260928 17:46:44.630756 5118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:44.630777 5118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3699861e864e159108b1f174ca3c4d, State: Initialized, Role: FOLLOWER
I20260928 17:46:44.630878 5118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3699861e864e159108b1f174ca3c4d" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36743 } }
I20260928 17:46:44.630942 5118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:44.630962 5118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:44.631006 5118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:44.631531 5118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3699861e864e159108b1f174ca3c4d" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36743 } }
I20260928 17:46:44.631630 5118 leader_election.cc:302] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b3699861e864e159108b1f174ca3c4d; no voters:
I20260928 17:46:44.631781 5118 leader_election.cc:288] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:44.631825 5123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:44.631968 5123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [term 1 LEADER]: Becoming Leader. State: Replica: 0b3699861e864e159108b1f174ca3c4d, State: Running, Role: LEADER
I20260928 17:46:44.632025 5118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:44.632077 5123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3699861e864e159108b1f174ca3c4d" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36743 } }
I20260928 17:46:44.632376 5123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b3699861e864e159108b1f174ca3c4d. Latest consensus state: current_term: 1 leader_uuid: "0b3699861e864e159108b1f174ca3c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3699861e864e159108b1f174ca3c4d" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36743 } } }
I20260928 17:46:44.632408 5124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b3699861e864e159108b1f174ca3c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3699861e864e159108b1f174ca3c4d" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36743 } } }
I20260928 17:46:44.632467 5123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:44.632522 5124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:44.632748 5127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:44.633249 5127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 17:46:44 dist-test-slave-w3bm krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:44.634881 5127 catalog_manager.cc:1347] Generated new cluster ID: 518cf2c72cfc44cdb6a21d02ee910245
I20260928 17:46:44.634928 5127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:46:44.637573 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:44.628213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.3.222.62:36743)
0928 17:46:44.628375 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:44.628382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:44.628412 (+ 30us) server_negotiation.cc:407] Connection header received
0928 17:46:44.628819 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:44.628825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:44.628870 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:44.628991 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:44.629587 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.630460 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.631366 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.631810 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.631909 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:44.634699 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:44.634703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:44.634704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:44.634714 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:44.636202 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.636506 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.636509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.636509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.636606 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.636775 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.636776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.636776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.636995 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:44.637144 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:44.637243 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 17:46:44.637366 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260928 17:46:44.664602 5127 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:44.665297 5127 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:46:44.670094 5127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d: Generated new TSK 0
I20260928 17:46:44.670252 5127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:46:44.706307 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36743
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:44.782246 5144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:44.782433 5144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:44.782464 5144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:44.783603 5144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:44.783641 5144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:44.783668 5144 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:46:44.783692 5144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:46:44.785054 5144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.222.62:36743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:44.785266 5144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:44.785447 5144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:44.789939 5149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:44.790004 5150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:44.789981 5144 server_base.cc:1034] running on GCE node
W20260928 17:46:44.789939 5152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:44.790369 5144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:44.790592 5144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:44.791720 5144 hybrid_clock.cc:648] HybridClock initialized: now 1790617604791704 us; error 28 us; skew 500 ppm
Sep 28 17:46:44 dist-test-slave-w3bm krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:44.799010 5144 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:46:44.799393 5144 webserver.cc:460] Webserver started at http://127.3.222.1:39495/ using document root <none> and password file <none>
I20260928 17:46:44.799568 5144 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:44.799605 5144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:44.799693 5144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:44.800608 5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "50792ffa6b0e4aaa9333a415c64b24b8"
format_stamp: "Formatted at 2026-09-28 17:46:44 on dist-test-slave-w3bm"
server_key: "aa8c66861028d03c75e43365763f58fc"
server_key_iv: "0966452e81b2f4ac7bd9d626b5a74b1a"
server_key_version: "encryptionkey@0"
I20260928 17:46:44.800892 5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "50792ffa6b0e4aaa9333a415c64b24b8"
format_stamp: "Formatted at 2026-09-28 17:46:44 on dist-test-slave-w3bm"
server_key: "aa8c66861028d03c75e43365763f58fc"
server_key_iv: "0966452e81b2f4ac7bd9d626b5a74b1a"
server_key_version: "encryptionkey@0"
I20260928 17:46:44.801981 5144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:44.802624 5159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:44.802773 5144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:44.802850 5144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "50792ffa6b0e4aaa9333a415c64b24b8"
format_stamp: "Formatted at 2026-09-28 17:46:44 on dist-test-slave-w3bm"
server_key: "aa8c66861028d03c75e43365763f58fc"
server_key_iv: "0966452e81b2f4ac7bd9d626b5a74b1a"
server_key_version: "encryptionkey@0"
I20260928 17:46:44.802897 5144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:44.818413 5144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:44.820227 5144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:44.820330 5144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:44.820751 5144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:44.820789 5144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:44.820811 5144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:44.820822 5144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:44.826932 5144 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:33261
I20260928 17:46:44.827006 5274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:33261 every 8 connection(s)
Sep 28 17:46:44 dist-test-slave-w3bm krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:44.827304 5144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:46:44.829859 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:44.821199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:46499 (local address 127.3.222.62:36743)
0928 17:46:44.821241 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:44.821243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:44.821448 (+ 205us) server_negotiation.cc:407] Connection header received
0928 17:46:44.821909 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:44.821910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:44.821923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:44.821974 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:44.822774 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.823552 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.824503 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.824808 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.824866 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:44.827727 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:44.827729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:44.827729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:44.827736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:44.828699 (+ 963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.829029 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.829030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.829031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.829085 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.829284 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.829285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.829285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.829389 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:44.829442 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:44.829679 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 17:46:44.829772 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:46:44.829972 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:44.821195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36743 (local address 127.3.222.1:46499)
0928 17:46:44.821343 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:44.821374 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:44.821813 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:44.822077 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:44.822082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:44.822402 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:44.822681 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:44.822687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.823638 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.823639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:44.824419 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:44.824421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.824912 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.824913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:44.825061 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:44.826010 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:44.826021 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:44.827655 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:44.828766 (+ 1111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:44.828772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:44.828774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:44.828964 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:44.829135 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:44.829136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:44.829136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:44.829233 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:44.829438 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:44.829444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:44.829571 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 17:46:44.829652 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 17:46:44.829784 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260928 17:46:44.830380 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 5144
I20260928 17:46:44.830415 5275 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36743
I20260928 17:46:44.830466 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:46:44.830504 5275 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:44.830631 3960 external_mini_cluster.cc:1428] Setting key 80a64cac3a02fa165fce194f5c1572d6
I20260928 17:46:44.830682 5275 heartbeater.cc:511] Master 127.3.222.62:36743 requested a full tablet report, sending...
I20260928 17:46:44.831424 5072 ts_manager.cc:194] Registered new tserver with Master: 50792ffa6b0e4aaa9333a415c64b24b8 (127.3.222.1:33261)
I20260928 17:46:44.831974 5072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:46499
I20260928 17:46:44.838018 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:44.832334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:39425 (local address 127.3.222.62:36743)
0928 17:46:44.832364 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:44.832366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:44.832904 (+ 538us) server_negotiation.cc:407] Connection header received
0928 17:46:44.832953 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:44.832954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:44.832964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:44.833017 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:44.833330 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.833851 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.834665 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.834932 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.834995 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:44.835845 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:44.835846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:44.835847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:44.835852 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:44.837012 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.837371 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.837372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.837373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.837427 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.837618 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.837619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.837619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.837712 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:44.837759 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:44.837900 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 17:46:44.837957 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:46:44.838068 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:44.832290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36743 (local address 127.3.222.1:39425)
0928 17:46:44.832843 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:44.832852 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:44.832902 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:44.833023 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:44.833023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:44.833218 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:44.833260 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:44.833262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.833932 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.833933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:44.834599 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:44.834601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.834992 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.834993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:44.835048 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:44.835437 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:44.835442 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:44.835764 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:44.837088 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:44.837089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:44.837089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:44.837282 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:44.837502 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:44.837503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:44.837503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:44.837565 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:44.837765 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:44.837766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:44.837811 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 17:46:44.837881 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 17:46:44.837969 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:46:44.855839 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36743
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:44.932528 5284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:44.932729 5284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:44.932760 5284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:44.933936 5284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:44.933972 5284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:44.934000 5284 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:46:44.934021 5284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:46:44.935405 5284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.222.62:36743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:44.935623 5284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:44.935806 5284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:44.940332 5289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:44.940318 5290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:44.940332 5292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:44.940663 5284 server_base.cc:1034] running on GCE node
I20260928 17:46:44.940874 5284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:44.941128 5284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:44.942281 5284 hybrid_clock.cc:648] HybridClock initialized: now 1790617604942263 us; error 32 us; skew 500 ppm
Sep 28 17:46:44 dist-test-slave-w3bm krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:44.944543 5284 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:46:44.944916 5284 webserver.cc:460] Webserver started at http://127.3.222.2:43023/ using document root <none> and password file <none>
I20260928 17:46:44.945107 5284 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:44.945147 5284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:44.945250 5284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:44.946194 5284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "0535c37002cc4a2d95ee25b70e9c184c"
format_stamp: "Formatted at 2026-09-28 17:46:44 on dist-test-slave-w3bm"
server_key: "5ed9a407657bc149bcb21f3275e93672"
server_key_iv: "1792da62d3c0212251ad8f10e5104bd3"
server_key_version: "encryptionkey@0"
I20260928 17:46:44.946475 5284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "0535c37002cc4a2d95ee25b70e9c184c"
format_stamp: "Formatted at 2026-09-28 17:46:44 on dist-test-slave-w3bm"
server_key: "5ed9a407657bc149bcb21f3275e93672"
server_key_iv: "1792da62d3c0212251ad8f10e5104bd3"
server_key_version: "encryptionkey@0"
I20260928 17:46:44.947705 5284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:44.948334 5299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:44.948457 5284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:44.948529 5284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "0535c37002cc4a2d95ee25b70e9c184c"
format_stamp: "Formatted at 2026-09-28 17:46:44 on dist-test-slave-w3bm"
server_key: "5ed9a407657bc149bcb21f3275e93672"
server_key_iv: "1792da62d3c0212251ad8f10e5104bd3"
server_key_version: "encryptionkey@0"
I20260928 17:46:44.948585 5284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:44.967686 5284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:44.969676 5284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:44.969807 5284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:44.970263 5284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:44.970302 5284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:44.970333 5284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:44.970350 5284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:44.976498 5284 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:40491
I20260928 17:46:44.976557 5414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:40491 every 8 connection(s)
Sep 28 17:46:44 dist-test-slave-w3bm krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:44.976920 5284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:46:44.979630 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:44.970722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:41495 (local address 127.3.222.62:36743)
0928 17:46:44.970786 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:44.970788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:44.971047 (+ 259us) server_negotiation.cc:407] Connection header received
0928 17:46:44.971591 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:44.971592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:44.971605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:44.971664 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:44.972455 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.973210 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.974688 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.974966 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.975043 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:44.977339 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:44.977341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:44.977341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:44.977350 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:44.978401 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.978756 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.978757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.978758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.978815 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.979026 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.979027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.979027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.979139 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:44.979195 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:44.979462 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 17:46:44.979542 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:46:44.979792 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:44.970710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36743 (local address 127.3.222.2:41495)
0928 17:46:44.970867 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:44.970908 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:44.971477 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:44.971760 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:44.971764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:44.972084 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:44.972356 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:44.972363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.973296 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.973297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:44.974610 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:44.974613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.975080 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.975081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:44.975200 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:44.975755 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:44.975764 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:44.977257 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:44.978471 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:44.978477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:44.978480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:44.978683 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:44.978866 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:44.978867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:44.978867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:44.978967 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:44.979191 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:44.979197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:44.979348 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 17:46:44.979435 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 17:46:44.979591 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260928 17:46:44.979990 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 5284
I20260928 17:46:44.980077 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:46:44.980249 3960 external_mini_cluster.cc:1428] Setting key 74f38e2d4f51eb63969835185fc31c58
I20260928 17:46:44.980331 5415 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36743
I20260928 17:46:44.980417 5415 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:44.980592 5415 heartbeater.cc:511] Master 127.3.222.62:36743 requested a full tablet report, sending...
I20260928 17:46:44.980902 5071 ts_manager.cc:194] Registered new tserver with Master: 0535c37002cc4a2d95ee25b70e9c184c (127.3.222.2:40491)
I20260928 17:46:44.981354 5071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:41495
I20260928 17:46:44.986809 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:44.981273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:35163 (local address 127.3.222.62:36743)
0928 17:46:44.981307 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:44.981309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:44.981420 (+ 111us) server_negotiation.cc:407] Connection header received
0928 17:46:44.981561 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:44.981562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:44.981572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:44.981616 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:44.981950 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.982479 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.983393 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.983572 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.983624 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:44.984563 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:44.984564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:44.984564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:44.984570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:44.985777 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.986123 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.986123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.986123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.986164 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:44.986328 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:44.986328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:44.986328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:44.986443 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:44.986486 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:44.986652 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 17:46:44.986722 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:46:44.986805 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:44.981222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36743 (local address 127.3.222.2:35163)
0928 17:46:44.981359 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:44.981368 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:44.981423 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:44.981620 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:44.981621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:44.981817 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:44.981873 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:44.981875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.982576 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.982577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:44.983325 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:44.983328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:44.983633 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:44.983634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:44.983700 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:44.984110 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:44.984115 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:44.984457 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:44.985847 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:44.985848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:44.985848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:44.986060 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:44.986212 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:44.986213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:44.986214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:44.986275 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:44.986492 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:44.986493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:44.986537 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 17:46:44.986608 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 17:46:44.986703 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:46:45.006055 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36743
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:45.082607 5424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:45.082791 5424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:45.082820 5424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:45.083971 5424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:45.084008 5424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:45.084034 5424 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:46:45.084059 5424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:46:45.085415 5424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.222.62:36743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:45.085636 5424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:45.085819 5424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:45.090700 5432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:45.090847 5424 server_base.cc:1034] running on GCE node
W20260928 17:46:45.090680 5429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:45.090688 5430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:45.091126 5424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:45.091367 5424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:45.092491 5424 hybrid_clock.cc:648] HybridClock initialized: now 1790617605092484 us; error 25 us; skew 500 ppm
Sep 28 17:46:45 dist-test-slave-w3bm krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:45.094714 5424 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:46:45.095134 5424 webserver.cc:460] Webserver started at http://127.3.222.3:45665/ using document root <none> and password file <none>
I20260928 17:46:45.095348 5424 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:45.095393 5424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:45.095511 5424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:45.096449 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e"
format_stamp: "Formatted at 2026-09-28 17:46:45 on dist-test-slave-w3bm"
server_key: "35020acbd88066c189d48b8fb3be6116"
server_key_iv: "5612d1a7408110043dac47245f6c81f1"
server_key_version: "encryptionkey@0"
I20260928 17:46:45.096753 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e"
format_stamp: "Formatted at 2026-09-28 17:46:45 on dist-test-slave-w3bm"
server_key: "35020acbd88066c189d48b8fb3be6116"
server_key_iv: "5612d1a7408110043dac47245f6c81f1"
server_key_version: "encryptionkey@0"
I20260928 17:46:45.097923 5424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:45.098580 5439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:45.098755 5424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:45.098848 5424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e"
format_stamp: "Formatted at 2026-09-28 17:46:45 on dist-test-slave-w3bm"
server_key: "35020acbd88066c189d48b8fb3be6116"
server_key_iv: "5612d1a7408110043dac47245f6c81f1"
server_key_version: "encryptionkey@0"
I20260928 17:46:45.098907 5424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:45.115828 5424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:45.117630 5424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:45.117758 5424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:45.118237 5424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:45.118278 5424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:45.118306 5424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:45.118323 5424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:45.124301 5424 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:33495
I20260928 17:46:45.124367 5554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:33495 every 8 connection(s)
Sep 28 17:46:45 dist-test-slave-w3bm krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:45.124631 5424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:46:45.127372 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.118674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:38589 (local address 127.3.222.62:36743)
0928 17:46:45.118714 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.118717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.118914 (+ 197us) server_negotiation.cc:407] Connection header received
0928 17:46:45.119465 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.119467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.119480 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.119560 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:45.120273 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.121039 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.122467 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.122756 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.122832 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.125077 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:45.125078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:45.125079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:45.125086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:45.126305 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.126640 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.126640 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.126640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.126685 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.126839 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.126839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.126839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.126926 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:45.126969 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.127236 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.127310 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:46:45.127599 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.118682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36743 (local address 127.3.222.3:38589)
0928 17:46:45.118808 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.118841 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.119369 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.119616 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.119620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.119915 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:45.120196 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.120202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.121141 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.121142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.122369 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.122372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.122830 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.122831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.122934 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.123398 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:45.123407 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:45.124985 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:45.126372 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:45.126377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:45.126379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:45.126586 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:45.126721 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:45.126721 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:45.126722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:45.126802 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:45.126966 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:45.126969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:45.127114 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.127197 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.127385 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
I20260928 17:46:45.128032 5555 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36743
I20260928 17:46:45.128108 5555 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:45.128273 5555 heartbeater.cc:511] Master 127.3.222.62:36743 requested a full tablet report, sending...
I20260928 17:46:45.128590 5072 ts_manager.cc:194] Registered new tserver with Master: 30bd7fc9b8e84857ad1e1a37cd93f89e (127.3.222.3:33495)
I20260928 17:46:45.129017 5072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:38589
I20260928 17:46:45.130425 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 5424
I20260928 17:46:45.130499 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:46:45.130653 3960 external_mini_cluster.cc:1428] Setting key 1f2820e1f2aa4ceba3fea1a599944b3c
I20260928 17:46:45.131381 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:46:45.134277 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.129295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:45379 (local address 127.3.222.62:36743)
0928 17:46:45.129331 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.129333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.129379 (+ 46us) server_negotiation.cc:407] Connection header received
0928 17:46:45.129433 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.129433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.129441 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.129483 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:45.129752 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.130213 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.130989 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.131186 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.131271 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.132024 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:45.132025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:45.132026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:45.132032 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:45.133276 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.133610 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.133611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.133611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.133658 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.133822 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.133823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.133824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.133948 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:45.134006 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.134134 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.134193 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:46:45.134284 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.129216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36743 (local address 127.3.222.3:45379)
0928 17:46:45.129333 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.129342 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.129392 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.129486 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.129486 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.129647 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:45.129691 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.129693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.130292 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.130293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.130923 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.130925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.131272 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.131273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.131347 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.131675 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:45.131678 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:45.131930 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:45.133342 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:45.133343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:45.133343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:45.133542 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:45.133705 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:45.133706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:45.133706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:45.133777 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:45.134004 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:45.134005 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:45.134047 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.134105 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.134194 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
Sep 28 17:46:45 dist-test-slave-w3bm krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:46:45 dist-test-slave-w3bm krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:45.160264 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.153312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.3.222.62:36743)
0928 17:46:45.153357 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.153359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.153366 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:46:45.153405 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.153406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.153417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.153479 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:45.154148 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.154877 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.155728 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.155985 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.156071 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.158168 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:45.158170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:45.158170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:45.158177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:45.159304 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.159609 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.159610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.159610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.159666 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.159820 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.159821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.159821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.159928 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:45.160016 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.160102 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.160174 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:46:45.161681 5072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:46:45.163196 5072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:45.169201 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.165154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:33261 (local address 127.0.0.1:51206)
0928 17:46:45.165431 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.165470 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.165553 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.165772 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.165775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.165776 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:45.165852 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.165854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.166933 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.166933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.168201 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.168204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.168947 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.168948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.169010 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.169018 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.169070 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.169128 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":3,"thread_start_us":50,"threads_started":1}
I20260928 17:46:45.169224 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.165242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.3.222.1:33261)
0928 17:46:45.165376 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.165381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.165635 (+ 254us) server_negotiation.cc:407] Connection header received
0928 17:46:45.165678 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.165681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.165698 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.165769 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:45.165939 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.166848 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.168297 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.168820 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.168865 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.168902 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.169104 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.169141 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260928 17:46:45.169492 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.165399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.3.222.3:33495)
0928 17:46:45.165536 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.165539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.165641 (+ 102us) server_negotiation.cc:407] Connection header received
0928 17:46:45.165710 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.165712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.165724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.165797 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:45.165955 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.167145 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.168429 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.169094 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.169144 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.169202 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.169371 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.169409 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260928 17:46:45.169629 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.165437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.3.222.2:40491)
0928 17:46:45.165580 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.165583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.165968 (+ 385us) server_negotiation.cc:407] Connection header received
0928 17:46:45.166008 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.166011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.166024 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.166082 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:45.166283 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.167371 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.168364 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.169070 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.169139 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.169194 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.169465 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.169533 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260928 17:46:45.169471 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.165155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:33495 (local address 127.0.0.1:53424)
0928 17:46:45.165590 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.165598 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.165649 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.165807 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.165808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.165810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:45.165877 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.165879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.167286 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.167287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.168365 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.168367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.169230 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.169231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.169294 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.169300 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.169350 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.169406 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":63,"thread_start_us":50,"threads_started":1}
I20260928 17:46:45.169766 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.165334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40491 (local address 127.0.0.1:43686)
0928 17:46:45.165848 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.165874 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.165918 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.166141 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.166142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.166144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:45.166186 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.166188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.167440 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.167441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.168294 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.168296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.169229 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.169231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.169294 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.169299 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.169345 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.169399 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":2,"thread_start_us":42,"threads_started":1}
I20260928 17:46:45.170207 5209 tablet_service.cc:1467] Processing CreateTablet for tablet e5e32a784e5f450f8b0e9323d0402ad0 (DEFAULT_TABLE table=test-table [id=40524d435252489e91418e64453fea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:45.170603 5209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e32a784e5f450f8b0e9323d0402ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:45.170584 5349 tablet_service.cc:1467] Processing CreateTablet for tablet e5e32a784e5f450f8b0e9323d0402ad0 (DEFAULT_TABLE table=test-table [id=40524d435252489e91418e64453fea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:45.170910 5349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e32a784e5f450f8b0e9323d0402ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:45.171799 5489 tablet_service.cc:1467] Processing CreateTablet for tablet e5e32a784e5f450f8b0e9323d0402ad0 (DEFAULT_TABLE table=test-table [id=40524d435252489e91418e64453fea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:45.172067 5489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e32a784e5f450f8b0e9323d0402ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:45.174118 5576 tablet_bootstrap.cc:492] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: Bootstrap starting.
I20260928 17:46:45.174320 5575 tablet_bootstrap.cc:492] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: Bootstrap starting.
I20260928 17:46:45.174346 5577 tablet_bootstrap.cc:492] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Bootstrap starting.
I20260928 17:46:45.174903 5575 tablet_bootstrap.cc:654] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:45.175143 5575 log.cc:824] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:45.175190 5577 tablet_bootstrap.cc:654] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:45.175563 5577 log.cc:824] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:45.175786 5576 tablet_bootstrap.cc:654] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:45.175904 5575 tablet_bootstrap.cc:492] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: No bootstrap required, opened a new log
I20260928 17:46:45.175971 5575 ts_tablet_manager.cc:1397] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:45.176143 5576 log.cc:824] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:45.176182 5577 tablet_bootstrap.cc:492] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: No bootstrap required, opened a new log
I20260928 17:46:45.176229 5577 ts_tablet_manager.cc:1397] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:45.176829 5576 tablet_bootstrap.cc:492] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: No bootstrap required, opened a new log
I20260928 17:46:45.176896 5576 ts_tablet_manager.cc:1397] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 17:46:45.177636 5577 raft_consensus.cc:348] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } }
I20260928 17:46:45.177760 5577 raft_consensus.cc:374] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:45.177789 5577 raft_consensus.cc:729] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30bd7fc9b8e84857ad1e1a37cd93f89e, State: Initialized, Role: FOLLOWER
I20260928 17:46:45.177789 5575 raft_consensus.cc:348] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } }
I20260928 17:46:45.177893 5577 consensus_queue.cc:260] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } }
I20260928 17:46:45.177963 5575 raft_consensus.cc:374] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:45.178025 5575 raft_consensus.cc:729] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50792ffa6b0e4aaa9333a415c64b24b8, State: Initialized, Role: FOLLOWER
I20260928 17:46:45.178171 5577 ts_tablet_manager.cc:1428] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:46:45.178162 5575 consensus_queue.cc:260] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } }
I20260928 17:46:45.178226 5555 heartbeater.cc:503] Master 127.3.222.62:36743 was elected leader, sending a full tablet report...
I20260928 17:46:45.178484 5575 ts_tablet_manager.cc:1428] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:46:45.178570 5275 heartbeater.cc:503] Master 127.3.222.62:36743 was elected leader, sending a full tablet report...
I20260928 17:46:45.178954 5576 raft_consensus.cc:348] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } }
I20260928 17:46:45.179126 5576 raft_consensus.cc:374] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:45.179189 5576 raft_consensus.cc:729] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0535c37002cc4a2d95ee25b70e9c184c, State: Initialized, Role: FOLLOWER
I20260928 17:46:45.179350 5576 consensus_queue.cc:260] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } }
I20260928 17:46:45.179597 5576 ts_tablet_manager.cc:1428] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 17:46:45.179621 5415 heartbeater.cc:503] Master 127.3.222.62:36743 was elected leader, sending a full tablet report...
W20260928 17:46:45.227564 5416 tablet.cc:2369] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:46:45.278831 5583 raft_consensus.cc:482] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:45.278940 5583 raft_consensus.cc:504] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } }
I20260928 17:46:45.279256 5583 leader_election.cc:288] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50792ffa6b0e4aaa9333a415c64b24b8 (127.3.222.1:33261), 30bd7fc9b8e84857ad1e1a37cd93f89e (127.3.222.3:33495)
I20260928 17:46:45.282291 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.279483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:51965 (local address 127.3.222.3:33495)
0928 17:46:45.279527 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.279529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.279541 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:46:45.279629 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.279631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.279642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.279699 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:45.279822 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.280558 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.281632 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.281976 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.282034 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.282045 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.282194 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.282229 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:46:45.282295 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.279568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:37559 (local address 127.3.222.1:33261)
0928 17:46:45.279608 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.279610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.279702 (+ 92us) server_negotiation.cc:407] Connection header received
0928 17:46:45.279789 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.279790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.279801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.279846 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:45.279994 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.280702 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.281625 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.281977 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.282035 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.282045 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.282187 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.282233 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:46:45.282291 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.279442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:33495 (local address 127.3.222.2:51965)
0928 17:46:45.279498 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.279509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.279551 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.279704 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.279705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.279707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:45.279770 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.279771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.280625 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.280625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.281555 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.281558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.282037 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.282038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.282128 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.282131 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.282173 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.282208 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260928 17:46:45.282303 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.279442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:33261 (local address 127.3.222.2:37559)
0928 17:46:45.279623 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.279633 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.279706 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.279852 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.279852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.279854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:45.279932 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.279934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.280771 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.280772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.281548 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.281550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.282049 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.282050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.282128 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.282132 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.282176 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.282215 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":25,"thread_start_us":109,"threads_started":1}
I20260928 17:46:45.282661 5509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e32a784e5f450f8b0e9323d0402ad0" candidate_uuid: "0535c37002cc4a2d95ee25b70e9c184c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" is_pre_election: true
I20260928 17:46:45.282799 5509 raft_consensus.cc:2454] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0535c37002cc4a2d95ee25b70e9c184c in term 0.
I20260928 17:46:45.282985 5229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e32a784e5f450f8b0e9323d0402ad0" candidate_uuid: "0535c37002cc4a2d95ee25b70e9c184c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" is_pre_election: true
I20260928 17:46:45.283020 5301 leader_election.cc:302] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0535c37002cc4a2d95ee25b70e9c184c, 30bd7fc9b8e84857ad1e1a37cd93f89e; no voters:
I20260928 17:46:45.283109 5229 raft_consensus.cc:2454] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0535c37002cc4a2d95ee25b70e9c184c in term 0.
I20260928 17:46:45.283121 5583 raft_consensus.cc:2781] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:45.283170 5583 raft_consensus.cc:482] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:45.283198 5583 raft_consensus.cc:3037] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:45.284119 5583 raft_consensus.cc:504] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } }
I20260928 17:46:45.284242 5583 leader_election.cc:288] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [CANDIDATE]: Term 1 election: Requested vote from peers 50792ffa6b0e4aaa9333a415c64b24b8 (127.3.222.1:33261), 30bd7fc9b8e84857ad1e1a37cd93f89e (127.3.222.3:33495)
I20260928 17:46:45.284376 5509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e32a784e5f450f8b0e9323d0402ad0" candidate_uuid: "0535c37002cc4a2d95ee25b70e9c184c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e"
I20260928 17:46:45.284446 5229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e32a784e5f450f8b0e9323d0402ad0" candidate_uuid: "0535c37002cc4a2d95ee25b70e9c184c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50792ffa6b0e4aaa9333a415c64b24b8"
I20260928 17:46:45.284456 5509 raft_consensus.cc:3037] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:45.284513 5229 raft_consensus.cc:3037] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:45.285094 5509 raft_consensus.cc:2454] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0535c37002cc4a2d95ee25b70e9c184c in term 1.
I20260928 17:46:45.285153 5229 raft_consensus.cc:2454] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0535c37002cc4a2d95ee25b70e9c184c in term 1.
I20260928 17:46:45.285240 5301 leader_election.cc:302] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0535c37002cc4a2d95ee25b70e9c184c, 30bd7fc9b8e84857ad1e1a37cd93f89e; no voters:
I20260928 17:46:45.285347 5583 raft_consensus.cc:2781] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:45.285492 5583 raft_consensus.cc:686] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 1 LEADER]: Becoming Leader. State: Replica: 0535c37002cc4a2d95ee25b70e9c184c, State: Running, Role: LEADER
I20260928 17:46:45.285574 5583 consensus_queue.cc:237] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } }
I20260928 17:46:45.286315 5072 catalog_manager.cc:5495] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0535c37002cc4a2d95ee25b70e9c184c (127.3.222.2). New cstate: current_term: 1 leader_uuid: "0535c37002cc4a2d95ee25b70e9c184c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } health_report { overall_health: HEALTHY } } }
I20260928 17:46:45.313663 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.311364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.3.222.2:40491)
0928 17:46:45.311420 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.311422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.311432 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:46:45.311521 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.311522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.311543 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.311595 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:45.311725 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.312173 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.312914 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.313098 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.313150 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.313336 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:45.313429 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:45.313494 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.313550 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.313570 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:46:45.315285 5509 raft_consensus.cc:1261] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Refusing update from remote peer 0535c37002cc4a2d95ee25b70e9c184c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:45.315285 5229 raft_consensus.cc:1261] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 FOLLOWER]: Refusing update from remote peer 0535c37002cc4a2d95ee25b70e9c184c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:45.315590 5585 consensus_queue.cc:1035] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [LEADER]: Connected to new peer: Peer: permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:45.315708 5583 consensus_queue.cc:1035] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:45.318356 5589 mvcc.cc:205] Tried to move back new op lower bound from 7334369711368568832 to 7334369711250112512. Current Snapshot: MvccSnapshot[applied={T|T < 7334369711368568832}]
I20260928 17:46:45.318756 5590 mvcc.cc:205] Tried to move back new op lower bound from 7334369711368568832 to 7334369711250112512. Current Snapshot: MvccSnapshot[applied={T|T < 7334369711368568832}]
I20260928 17:46:45.320958 5071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:45.321043 5071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:45.321698 5071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d: Sending DeleteTablet for 3 replicas of tablet e5e32a784e5f450f8b0e9323d0402ad0
I20260928 17:46:45.322018 5489 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e32a784e5f450f8b0e9323d0402ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:45 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:53424
I20260928 17:46:45.322024 5349 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e32a784e5f450f8b0e9323d0402ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:45 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:43686
I20260928 17:46:45.322170 5071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:46:45.322300 5071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:45.322279 5209 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e32a784e5f450f8b0e9323d0402ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:45 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:51206
I20260928 17:46:45.322453 5599 tablet_replica.cc:330] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: stopping tablet replica
I20260928 17:46:45.322564 5598 tablet_replica.cc:330] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: stopping tablet replica
I20260928 17:46:45.322577 5599 raft_consensus.cc:2229] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:45.322669 5599 raft_consensus.cc:2258] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:45.322710 5598 raft_consensus.cc:2229] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:46:45.322861 5598 raft_consensus.cc:2258] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:45.323258 5599 ts_tablet_manager.cc:1905] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:45.323414 5209 tablet_service.cc:1467] Processing CreateTablet for tablet 14191f81aab3404fa2637c5c95c0d803 (DEFAULT_TABLE table=test-table [id=30b65f2f077a49b5babfd594f0879a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:45.323493 5349 tablet_service.cc:1467] Processing CreateTablet for tablet 14191f81aab3404fa2637c5c95c0d803 (DEFAULT_TABLE table=test-table [id=30b65f2f077a49b5babfd594f0879a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:45.323525 5209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14191f81aab3404fa2637c5c95c0d803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:45.323572 5349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14191f81aab3404fa2637c5c95c0d803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:45.324888 5489 tablet_service.cc:1467] Processing CreateTablet for tablet 14191f81aab3404fa2637c5c95c0d803 (DEFAULT_TABLE table=test-table [id=30b65f2f077a49b5babfd594f0879a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:45.324975 5489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14191f81aab3404fa2637c5c95c0d803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:45.325273 5576 tablet_bootstrap.cc:492] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: Bootstrap starting.
I20260928 17:46:45.325646 5576 tablet_bootstrap.cc:654] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:45.325959 5599 ts_tablet_manager.cc:1918] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:46:45.326089 5599 log.cc:1196] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/e5e32a784e5f450f8b0e9323d0402ad0
I20260928 17:46:45.326156 5598 ts_tablet_manager.cc:1905] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:45.326182 5577 tablet_bootstrap.cc:492] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Bootstrap starting.
I20260928 17:46:45.326205 5575 tablet_bootstrap.cc:492] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: Bootstrap starting.
I20260928 17:46:45.326426 5599 ts_tablet_manager.cc:1939] T e5e32a784e5f450f8b0e9323d0402ad0 P 50792ffa6b0e4aaa9333a415c64b24b8: Deleting consensus metadata
I20260928 17:46:45.326540 5575 tablet_bootstrap.cc:654] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:45.326694 5577 tablet_bootstrap.cc:654] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:45.326844 5056 catalog_manager.cc:4841] TS 50792ffa6b0e4aaa9333a415c64b24b8 (127.3.222.1:33261): tablet e5e32a784e5f450f8b0e9323d0402ad0 (table test-table [id=40524d435252489e91418e64453fea2a]) successfully deleted
I20260928 17:46:45.327346 5576 tablet_bootstrap.cc:492] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: No bootstrap required, opened a new log
I20260928 17:46:45.327390 5576 ts_tablet_manager.cc:1397] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:45.327534 5576 raft_consensus.cc:348] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.327594 5576 raft_consensus.cc:374] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:45.327613 5576 raft_consensus.cc:729] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0535c37002cc4a2d95ee25b70e9c184c, State: Initialized, Role: FOLLOWER
I20260928 17:46:45.327659 5575 tablet_bootstrap.cc:492] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: No bootstrap required, opened a new log
I20260928 17:46:45.327651 5576 consensus_queue.cc:260] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.327693 5598 ts_tablet_manager.cc:1918] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:46:45.327710 5575 ts_tablet_manager.cc:1397] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:45.327725 5576 ts_tablet_manager.cc:1428] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:45.327745 5598 log.cc:1196] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/e5e32a784e5f450f8b0e9323d0402ad0
I20260928 17:46:45.327836 5575 raft_consensus.cc:348] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.327903 5575 raft_consensus.cc:374] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:45.327922 5575 raft_consensus.cc:729] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50792ffa6b0e4aaa9333a415c64b24b8, State: Initialized, Role: FOLLOWER
I20260928 17:46:45.327956 5575 consensus_queue.cc:260] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.328025 5575 ts_tablet_manager.cc:1428] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:45.328202 5597 tablet_replica.cc:330] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: stopping tablet replica
I20260928 17:46:45.328320 5597 raft_consensus.cc:2229] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:45.328454 5597 raft_consensus.cc:2258] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:45.328733 5598 ts_tablet_manager.cc:1939] T e5e32a784e5f450f8b0e9323d0402ad0 P 0535c37002cc4a2d95ee25b70e9c184c: Deleting consensus metadata
I20260928 17:46:45.329092 5597 ts_tablet_manager.cc:1905] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:45.329118 5056 catalog_manager.cc:4841] TS 0535c37002cc4a2d95ee25b70e9c184c (127.3.222.2:40491): tablet e5e32a784e5f450f8b0e9323d0402ad0 (table test-table [id=40524d435252489e91418e64453fea2a]) successfully deleted
I20260928 17:46:45.330245 5597 ts_tablet_manager.cc:1918] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:46:45.330302 5597 log.cc:1196] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/e5e32a784e5f450f8b0e9323d0402ad0
I20260928 17:46:45.330447 5597 ts_tablet_manager.cc:1939] T e5e32a784e5f450f8b0e9323d0402ad0 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Deleting consensus metadata
I20260928 17:46:45.330814 5057 catalog_manager.cc:4841] TS 30bd7fc9b8e84857ad1e1a37cd93f89e (127.3.222.3:33495): tablet e5e32a784e5f450f8b0e9323d0402ad0 (table test-table [id=40524d435252489e91418e64453fea2a]) successfully deleted
I20260928 17:46:45.331527 5577 tablet_bootstrap.cc:492] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: No bootstrap required, opened a new log
I20260928 17:46:45.331564 5577 ts_tablet_manager.cc:1397] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 17:46:45.331664 5577 raft_consensus.cc:348] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.331720 5577 raft_consensus.cc:374] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:45.331733 5577 raft_consensus.cc:729] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30bd7fc9b8e84857ad1e1a37cd93f89e, State: Initialized, Role: FOLLOWER
I20260928 17:46:45.331761 5577 consensus_queue.cc:260] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.331836 5577 ts_tablet_manager.cc:1428] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260928 17:46:45.375123 5556 tablet.cc:2369] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:46:45.532998 5582 raft_consensus.cc:482] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:45.533100 5582 raft_consensus.cc:504] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.533435 5582 leader_election.cc:288] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0535c37002cc4a2d95ee25b70e9c184c (127.3.222.2:40491), 30bd7fc9b8e84857ad1e1a37cd93f89e (127.3.222.3:33495)
I20260928 17:46:45.536216 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.533686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:45155 (local address 127.3.222.3:33495)
0928 17:46:45.533726 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.533727 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.533828 (+ 101us) server_negotiation.cc:407] Connection header received
0928 17:46:45.533889 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.533890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.533897 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.533943 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:45.534086 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.534562 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.535589 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.535919 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.535971 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.535981 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.536113 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.536155 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:46:45.536221 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.533653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:34867 (local address 127.3.222.2:40491)
0928 17:46:45.533718 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.533720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.533812 (+ 92us) server_negotiation.cc:407] Connection header received
0928 17:46:45.533869 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.533870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.533878 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.533929 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:45.534075 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.534597 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.535585 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.535928 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.535988 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.535998 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.536113 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.536161 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:46:45.536221 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.533609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:33495 (local address 127.3.222.1:45155)
0928 17:46:45.533761 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.533770 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.533829 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.533957 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.533958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.533960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:45.534029 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.534031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.534631 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.534631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.535503 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.535506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.535983 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.535984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.536048 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.536052 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.536094 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.536134 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":29,"thread_start_us":71,"threads_started":1}
I20260928 17:46:45.536229 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.533600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40491 (local address 127.3.222.1:34867)
0928 17:46:45.533740 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.533751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.533813 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.533938 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.533939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.533941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:45.533998 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.534000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.534680 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.534681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.535508 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.535510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.535997 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.535998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.536054 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.536057 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.536102 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.536140 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260928 17:46:45.536466 5509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14191f81aab3404fa2637c5c95c0d803" candidate_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" is_pre_election: true
I20260928 17:46:45.536502 5369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14191f81aab3404fa2637c5c95c0d803" candidate_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0535c37002cc4a2d95ee25b70e9c184c" is_pre_election: true
I20260928 17:46:45.536580 5509 raft_consensus.cc:2454] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50792ffa6b0e4aaa9333a415c64b24b8 in term 0.
I20260928 17:46:45.536607 5369 raft_consensus.cc:2454] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50792ffa6b0e4aaa9333a415c64b24b8 in term 0.
I20260928 17:46:45.536793 5160 leader_election.cc:302] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0535c37002cc4a2d95ee25b70e9c184c, 50792ffa6b0e4aaa9333a415c64b24b8; no voters:
I20260928 17:46:45.536926 5582 raft_consensus.cc:2781] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:45.536970 5582 raft_consensus.cc:482] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:45.536983 5582 raft_consensus.cc:3037] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:45.537626 5582 raft_consensus.cc:504] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.537735 5582 leader_election.cc:288] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [CANDIDATE]: Term 1 election: Requested vote from peers 0535c37002cc4a2d95ee25b70e9c184c (127.3.222.2:40491), 30bd7fc9b8e84857ad1e1a37cd93f89e (127.3.222.3:33495)
I20260928 17:46:45.537882 5369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14191f81aab3404fa2637c5c95c0d803" candidate_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0535c37002cc4a2d95ee25b70e9c184c"
I20260928 17:46:45.537951 5369 raft_consensus.cc:3037] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:45.537972 5509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14191f81aab3404fa2637c5c95c0d803" candidate_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e"
I20260928 17:46:45.538058 5509 raft_consensus.cc:3037] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:45.538573 5369 raft_consensus.cc:2454] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50792ffa6b0e4aaa9333a415c64b24b8 in term 1.
I20260928 17:46:45.538648 5509 raft_consensus.cc:2454] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50792ffa6b0e4aaa9333a415c64b24b8 in term 1.
I20260928 17:46:45.538795 5161 leader_election.cc:302] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0535c37002cc4a2d95ee25b70e9c184c, 30bd7fc9b8e84857ad1e1a37cd93f89e, 50792ffa6b0e4aaa9333a415c64b24b8; no voters:
I20260928 17:46:45.538872 5582 raft_consensus.cc:2781] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:45.538968 5582 raft_consensus.cc:686] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 LEADER]: Becoming Leader. State: Replica: 50792ffa6b0e4aaa9333a415c64b24b8, State: Running, Role: LEADER
I20260928 17:46:45.539022 5582 consensus_queue.cc:237] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.539669 5072 catalog_manager.cc:5495] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50792ffa6b0e4aaa9333a415c64b24b8 (127.3.222.1). New cstate: current_term: 1 leader_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } health_report { overall_health: UNKNOWN } } }
W20260928 17:46:45.578156 5276 tablet.cc:2369] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 17:46:45 dist-test-slave-w3bm krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:45.590037 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.583847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36743 (local address 127.0.0.1:46280)
0928 17:46:45.584031 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.584039 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.584102 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.584319 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.584320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.584612 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:45.584659 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.584660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.585320 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.585322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.585866 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.585867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.586176 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.586176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.586251 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.586736 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:45.586740 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:45.588049 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:45.589243 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:45.589250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:45.589251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:45.589425 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:45.589561 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:45.589562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:45.589562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:45.589608 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:45.589772 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:45.589776 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:45.589812 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.589857 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.589941 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
I20260928 17:46:45.590078 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.583912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.3.222.62:36743)
0928 17:46:45.583973 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.583975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.584143 (+ 168us) server_negotiation.cc:407] Connection header received
0928 17:46:45.584171 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.584172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.584181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.584239 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:45.584728 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.585221 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.585938 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.586127 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.586174 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.588135 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:45.588137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:45.588137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:45.588142 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:45.589176 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.589477 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.589478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.589478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.589519 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.589645 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.589645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.589645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.589731 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:45.589779 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.589878 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.589981 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":37}
I20260928 17:46:45.591794 5072 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:46280:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 17:46:45.591965 5072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:45.593109 5349 tablet_service.cc:1467] Processing CreateTablet for tablet 4d74a435d26a434f90e4862ae404627e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e17c7d27467b4dfb9fffbebd182b8161]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:46:45.593119 5209 tablet_service.cc:1467] Processing CreateTablet for tablet 4d74a435d26a434f90e4862ae404627e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e17c7d27467b4dfb9fffbebd182b8161]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:46:45.593202 5489 tablet_service.cc:1467] Processing CreateTablet for tablet 4d74a435d26a434f90e4862ae404627e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e17c7d27467b4dfb9fffbebd182b8161]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:46:45.593261 5209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d74a435d26a434f90e4862ae404627e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:45.593282 5349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d74a435d26a434f90e4862ae404627e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:45.593322 5489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d74a435d26a434f90e4862ae404627e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:45.594277 5577 tablet_bootstrap.cc:492] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: Bootstrap starting.
I20260928 17:46:45.594637 5576 tablet_bootstrap.cc:492] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: Bootstrap starting.
I20260928 17:46:45.594661 5577 tablet_bootstrap.cc:654] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:45.594858 5575 tablet_bootstrap.cc:492] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: Bootstrap starting.
I20260928 17:46:45.594952 5576 tablet_bootstrap.cc:654] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:45.595184 5575 tablet_bootstrap.cc:654] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:45.595297 5577 tablet_bootstrap.cc:492] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: No bootstrap required, opened a new log
I20260928 17:46:45.595336 5577 ts_tablet_manager.cc:1397] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:45.595484 5577 raft_consensus.cc:348] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.595558 5577 raft_consensus.cc:374] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:45.595548 5576 tablet_bootstrap.cc:492] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: No bootstrap required, opened a new log
I20260928 17:46:45.595579 5577 raft_consensus.cc:729] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30bd7fc9b8e84857ad1e1a37cd93f89e, State: Initialized, Role: FOLLOWER
I20260928 17:46:45.595623 5576 ts_tablet_manager.cc:1397] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:46:45.595620 5577 consensus_queue.cc:260] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.595697 5577 ts_tablet_manager.cc:1428] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:45.595738 5576 raft_consensus.cc:348] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.595808 5576 raft_consensus.cc:374] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:45.595825 5576 raft_consensus.cc:729] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0535c37002cc4a2d95ee25b70e9c184c, State: Initialized, Role: FOLLOWER
I20260928 17:46:45.595826 5581 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.595860 5576 consensus_queue.cc:260] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.595875 5575 tablet_bootstrap.cc:492] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: No bootstrap required, opened a new log
I20260928 17:46:45.595898 5581 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:45.595911 5575 ts_tablet_manager.cc:1397] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:45.595939 5576 ts_tablet_manager.cc:1428] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:45.595956 5585 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.596016 5585 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:45.596028 5575 raft_consensus.cc:348] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.596096 5575 raft_consensus.cc:374] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:45.596122 5575 raft_consensus.cc:729] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50792ffa6b0e4aaa9333a415c64b24b8, State: Initialized, Role: FOLLOWER
I20260928 17:46:45.596158 5575 consensus_queue.cc:260] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.596233 5575 ts_tablet_manager.cc:1428] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:45.596297 5582 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.596374 5582 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:45.636533 5582 raft_consensus.cc:482] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:45.636617 5582 raft_consensus.cc:504] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.636740 5582 leader_election.cc:288] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0535c37002cc4a2d95ee25b70e9c184c (127.3.222.2:40491), 30bd7fc9b8e84857ad1e1a37cd93f89e (127.3.222.3:33495)
I20260928 17:46:45.636926 5509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d74a435d26a434f90e4862ae404627e" candidate_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" is_pre_election: true
I20260928 17:46:45.636935 5369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d74a435d26a434f90e4862ae404627e" candidate_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0535c37002cc4a2d95ee25b70e9c184c" is_pre_election: true
I20260928 17:46:45.637002 5509 raft_consensus.cc:2454] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50792ffa6b0e4aaa9333a415c64b24b8 in term 0.
I20260928 17:46:45.637024 5369 raft_consensus.cc:2454] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50792ffa6b0e4aaa9333a415c64b24b8 in term 0.
I20260928 17:46:45.637117 5161 leader_election.cc:302] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30bd7fc9b8e84857ad1e1a37cd93f89e, 50792ffa6b0e4aaa9333a415c64b24b8; no voters:
I20260928 17:46:45.637247 5582 raft_consensus.cc:2781] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:45.637287 5582 raft_consensus.cc:482] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:45.637306 5582 raft_consensus.cc:3037] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:45.637907 5582 raft_consensus.cc:504] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.638016 5582 leader_election.cc:288] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [CANDIDATE]: Term 1 election: Requested vote from peers 0535c37002cc4a2d95ee25b70e9c184c (127.3.222.2:40491), 30bd7fc9b8e84857ad1e1a37cd93f89e (127.3.222.3:33495)
I20260928 17:46:45.638161 5509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d74a435d26a434f90e4862ae404627e" candidate_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e"
I20260928 17:46:45.638236 5509 raft_consensus.cc:3037] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:45.638743 5509 raft_consensus.cc:2454] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50792ffa6b0e4aaa9333a415c64b24b8 in term 1.
I20260928 17:46:45.638839 5369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d74a435d26a434f90e4862ae404627e" candidate_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0535c37002cc4a2d95ee25b70e9c184c"
I20260928 17:46:45.638882 5161 leader_election.cc:302] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30bd7fc9b8e84857ad1e1a37cd93f89e, 50792ffa6b0e4aaa9333a415c64b24b8; no voters:
I20260928 17:46:45.638913 5369 raft_consensus.cc:3037] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:45.639001 5582 raft_consensus.cc:2781] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:45.639060 5582 raft_consensus.cc:686] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 LEADER]: Becoming Leader. State: Replica: 50792ffa6b0e4aaa9333a415c64b24b8, State: Running, Role: LEADER
I20260928 17:46:45.639099 5582 consensus_queue.cc:237] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } }
I20260928 17:46:45.639415 5605 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: TxnStatusTablet state changed. Reason: New leader 50792ffa6b0e4aaa9333a415c64b24b8. Latest consensus state: current_term: 1 leader_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.639505 5605 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:46:45.639655 5369 raft_consensus.cc:2454] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50792ffa6b0e4aaa9333a415c64b24b8 in term 1.
I20260928 17:46:45.639899 5072 catalog_manager.cc:5495] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50792ffa6b0e4aaa9333a415c64b24b8 (127.3.222.1). New cstate: current_term: 1 leader_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } health_report { overall_health: UNKNOWN } } }
I20260928 17:46:45.640169 5617 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 17:46:45.640269 5617 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 17:46:45.655992 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.653060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:33261 (local address 127.0.0.1:51212)
0928 17:46:45.653123 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.653134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.653222 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.653368 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.653369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.653370 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:46:45.653428 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.653431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.654335 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.654335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.654963 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.654965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.655438 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.655439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.655493 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.655508 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:46:45.655817 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:46:45.655828 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.655885 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.655927 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 17:46:45.655995 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.653120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.3.222.1:33261)
0928 17:46:45.653163 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.653165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.653201 (+ 36us) server_negotiation.cc:407] Connection header received
0928 17:46:45.653288 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.653289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.653307 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.653360 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:45.653496 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.654265 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.655036 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.655369 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.655433 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.655601 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:45.655740 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:45.655808 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.655899 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.655921 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:46:45.656927 5509 raft_consensus.cc:1261] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Refusing update from remote peer 50792ffa6b0e4aaa9333a415c64b24b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:45.657186 5369 raft_consensus.cc:1261] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c [term 1 FOLLOWER]: Refusing update from remote peer 50792ffa6b0e4aaa9333a415c64b24b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:45.657184 5605 consensus_queue.cc:1035] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:45.657635 5581 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: TxnStatusTablet state changed. Reason: New leader 50792ffa6b0e4aaa9333a415c64b24b8. Latest consensus state: current_term: 1 leader_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.657719 5581 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:45.657768 5605 consensus_queue.cc:1035] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:45.658165 5605 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.658234 5605 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:46:45.658500 5585 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: TxnStatusTablet state changed. Reason: New leader 50792ffa6b0e4aaa9333a415c64b24b8. Latest consensus state: current_term: 1 leader_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.658586 5585 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:45.658587 5581 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.658656 5581 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 30bd7fc9b8e84857ad1e1a37cd93f89e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:45.660241 5605 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.660315 5605 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 50792ffa6b0e4aaa9333a415c64b24b8: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:46:45.660650 5585 tablet_replica.cc:439] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50792ffa6b0e4aaa9333a415c64b24b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33261 } } peers { permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 } } peers { permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 } } }
I20260928 17:46:45.660737 5585 tablet_replica.cc:442] T 4d74a435d26a434f90e4862ae404627e P 0535c37002cc4a2d95ee25b70e9c184c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:45.660779 5588 mvcc.cc:205] Tried to move back new op lower bound from 7334369712768831488 to 7334369712697999360. Current Snapshot: MvccSnapshot[applied={T|T < 7334369712768831488 or (T in {7334369712768831488})}]
I20260928 17:46:45.662711 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.660094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.3.222.1:33261)
0928 17:46:45.660144 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.660146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.660154 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:46:45.660172 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.660173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.660183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.660231 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:45.660491 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.661155 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.662175 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.662346 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.662396 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.662534 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:45.662571 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:45.662601 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.662653 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.662667 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:46:45.667466 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.664888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:43307 (local address 127.3.222.1:33261)
0928 17:46:45.664920 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.664921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.665116 (+ 195us) server_negotiation.cc:407] Connection header received
0928 17:46:45.665134 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.665134 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.665142 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.665175 (+ 33us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:45.665280 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.665700 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.666770 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.666934 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.666974 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.667199 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:45.667267 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:45.667324 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.667385 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.667400 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:46:45.667558 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.664854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:33261 (local address 127.3.222.1:43307)
0928 17:46:45.665003 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:45.665014 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:45.665073 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:45.665178 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:45.665179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:45.665180 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:46:45.665230 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.665231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.665754 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.665755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.666686 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:45.666688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.666987 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.666988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:45.667073 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.667088 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:46:45.667332 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:46:45.667334 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:46:45.667382 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 17:46:45.667422 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260928 17:46:45.669490 5369 raft_consensus.cc:1261] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [term 1 FOLLOWER]: Refusing update from remote peer 50792ffa6b0e4aaa9333a415c64b24b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:45.669504 5509 raft_consensus.cc:1261] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Refusing update from remote peer 50792ffa6b0e4aaa9333a415c64b24b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:45.669709 5625 consensus_queue.cc:1035] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0535c37002cc4a2d95ee25b70e9c184c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:45.669786 5582 consensus_queue.cc:1035] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd7fc9b8e84857ad1e1a37cd93f89e" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:45.680467 5070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:45.680549 5070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:45.681078 5070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b3699861e864e159108b1f174ca3c4d: Sending DeleteTablet for 3 replicas of tablet 14191f81aab3404fa2637c5c95c0d803
I20260928 17:46:45.681303 5489 tablet_service.cc:1514] Processing DeleteTablet for tablet 14191f81aab3404fa2637c5c95c0d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:45 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:53424
I20260928 17:46:45.681342 5208 tablet_service.cc:1514] Processing DeleteTablet for tablet 14191f81aab3404fa2637c5c95c0d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:45 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:51206
I20260928 17:46:45.681392 5349 tablet_service.cc:1514] Processing DeleteTablet for tablet 14191f81aab3404fa2637c5c95c0d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:45 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:43686
I20260928 17:46:45.681461 5599 tablet_replica.cc:330] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: stopping tablet replica
I20260928 17:46:45.681493 5598 tablet_replica.cc:330] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: stopping tablet replica
I20260928 17:46:45.681496 5597 tablet_replica.cc:330] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: stopping tablet replica
I20260928 17:46:45.681509 5599 raft_consensus.cc:2229] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:46:45.681542 5598 raft_consensus.cc:2229] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:45.681555 5597 raft_consensus.cc:2229] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:45.681571 5598 raft_consensus.cc:2258] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:45.681581 5599 raft_consensus.cc:2258] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:45.681587 5597 raft_consensus.cc:2258] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:45.681816 5598 ts_tablet_manager.cc:1905] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:45.681900 5597 ts_tablet_manager.cc:1905] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:45.682143 5599 ts_tablet_manager.cc:1905] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:45.683794 5599 ts_tablet_manager.cc:1918] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:46:45.683804 5598 ts_tablet_manager.cc:1918] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:46:45.683847 5599 log.cc:1196] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/14191f81aab3404fa2637c5c95c0d803
I20260928 17:46:45.683852 5598 log.cc:1196] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/14191f81aab3404fa2637c5c95c0d803
I20260928 17:46:45.684056 5598 ts_tablet_manager.cc:1939] T 14191f81aab3404fa2637c5c95c0d803 P 0535c37002cc4a2d95ee25b70e9c184c: Deleting consensus metadata
I20260928 17:46:45.684054 5599 ts_tablet_manager.cc:1939] T 14191f81aab3404fa2637c5c95c0d803 P 50792ffa6b0e4aaa9333a415c64b24b8: Deleting consensus metadata
I20260928 17:46:45.684333 5597 ts_tablet_manager.cc:1918] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:46:45.684376 5597 log.cc:1196] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/14191f81aab3404fa2637c5c95c0d803
I20260928 17:46:45.684377 5056 catalog_manager.cc:4841] TS 50792ffa6b0e4aaa9333a415c64b24b8 (127.3.222.1:33261): tablet 14191f81aab3404fa2637c5c95c0d803 (table test-table [id=30b65f2f077a49b5babfd594f0879a14]) successfully deleted
I20260928 17:46:45.684437 5056 catalog_manager.cc:4841] TS 0535c37002cc4a2d95ee25b70e9c184c (127.3.222.2:40491): tablet 14191f81aab3404fa2637c5c95c0d803 (table test-table [id=30b65f2f077a49b5babfd594f0879a14]) successfully deleted
I20260928 17:46:45.684551 5597 ts_tablet_manager.cc:1939] T 14191f81aab3404fa2637c5c95c0d803 P 30bd7fc9b8e84857ad1e1a37cd93f89e: Deleting consensus metadata
I20260928 17:46:45.686903 5057 catalog_manager.cc:4841] TS 30bd7fc9b8e84857ad1e1a37cd93f89e (127.3.222.3:33495): tablet 14191f81aab3404fa2637c5c95c0d803 (table test-table [id=30b65f2f077a49b5babfd594f0879a14]) successfully deleted
Sep 28 17:46:45 dist-test-slave-w3bm krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:46:45.691650 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.683511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.3.222.1:33261)
0928 17:46:45.683547 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.683549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.683682 (+ 133us) server_negotiation.cc:407] Connection header received
0928 17:46:45.683757 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.683758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.683768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.683796 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:45.684505 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.685239 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.685928 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.686222 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.686286 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.689123 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:45.689129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:45.689130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:45.689136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:45.690585 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.690935 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.690937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.690937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.690984 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.691153 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.691153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.691153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.691336 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:45.691447 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.691537 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.691592 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:46:45.691993 5269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51232
I20260928 17:46:45.697880 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:45.692850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.3.222.62:36743)
0928 17:46:45.692896 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:45.692898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:45.692906 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:46:45.692960 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:45.692961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:45.692971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:45.693013 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:45.693451 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.693929 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.694556 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:45.694762 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:45.694815 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:45.695790 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:45.695792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:45.695792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:45.695800 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:45.696960 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.697236 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.697237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.697237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.697299 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:45.697450 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:45.697450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:45.697451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:45.697553 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:45.697632 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:45.697729 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 17:46:45.697795 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 17:46:45.698066 5070 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46282
I20260928 17:46:45.698573 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 5144
I20260928 17:46:45.704443 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 5284
I20260928 17:46:45.709235 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 5424
I20260928 17:46:45.712803 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 5040
2026-09-28T17:46:45Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2266 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:46:45 dist-test-slave-w3bm krb5kdc[5635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:46:45 dist-test-slave-w3bm krb5kdc[5635](info): set up 2 sockets
Sep 28 17:46:45 dist-test-slave-w3bm krb5kdc[5635](info): commencing operation
krb5kdc: starting...
W20260928 17:46:46.666615 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:46:46 dist-test-slave-w3bm krb5kdc[5635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:46:46.777779 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35357
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:46.854517 5648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:46.854704 5648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:46.854724 5648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:46.855794 5648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:46.855825 5648 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 17:46:46.855837 5648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:46.855849 5648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:46.855865 5648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:46.855877 5648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:46.857230 5648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:35357
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:46.857412 5648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:46.857561 5648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:46.861907 5654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:46.861976 5653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:46.861994 5656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:46.862107 5648 server_base.cc:1034] running on GCE node
I20260928 17:46:46.862413 5648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:46.862699 5648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:46.863838 5648 hybrid_clock.cc:648] HybridClock initialized: now 1790617606863824 us; error 36 us; skew 500 ppm
Sep 28 17:46:46 dist-test-slave-w3bm krb5kdc[5635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:46.865978 5648 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:46:46.866343 5648 webserver.cc:460] Webserver started at http://127.3.222.62:46883/ using document root <none> and password file <none>
I20260928 17:46:46.866533 5648 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:46.866580 5648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:46.866698 5648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:46.867688 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "b5faff593dc64d52a7634a9aa96d6298"
format_stamp: "Formatted at 2026-09-28 17:46:46 on dist-test-slave-w3bm"
server_key: "6ef0c2ed92cdfaafa54b86dcd0ced898"
server_key_iv: "cfe5df9ae1320e28c1102e629407539f"
server_key_version: "encryptionkey@0"
I20260928 17:46:46.867985 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "b5faff593dc64d52a7634a9aa96d6298"
format_stamp: "Formatted at 2026-09-28 17:46:46 on dist-test-slave-w3bm"
server_key: "6ef0c2ed92cdfaafa54b86dcd0ced898"
server_key_iv: "cfe5df9ae1320e28c1102e629407539f"
server_key_version: "encryptionkey@0"
I20260928 17:46:46.869107 5648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:46.869720 5663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:46.869858 5648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:46.869933 5648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "b5faff593dc64d52a7634a9aa96d6298"
format_stamp: "Formatted at 2026-09-28 17:46:46 on dist-test-slave-w3bm"
server_key: "6ef0c2ed92cdfaafa54b86dcd0ced898"
server_key_iv: "cfe5df9ae1320e28c1102e629407539f"
server_key_version: "encryptionkey@0"
I20260928 17:46:46.869990 5648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:46.882287 5648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:46.884063 5648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:46.884177 5648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:46.888181 5648 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:35357
I20260928 17:46:46.888257 5725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:35357 every 8 connection(s)
I20260928 17:46:46.888502 5648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:46.889055 5726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:46.891105 5726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298: Bootstrap starting.
I20260928 17:46:46.891593 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 5648
I20260928 17:46:46.891649 5726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:46.891669 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:46.891812 3960 external_mini_cluster.cc:1428] Setting key 44dae8c7b8e7d0858f61acf6fae4f2b2
I20260928 17:46:46.891899 5726 log.cc:824] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:46.892580 5726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298: No bootstrap required, opened a new log
I20260928 17:46:46.894362 5726 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5faff593dc64d52a7634a9aa96d6298" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35357 } }
I20260928 17:46:46.894495 5726 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:46.894526 5726 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5faff593dc64d52a7634a9aa96d6298, State: Initialized, Role: FOLLOWER
I20260928 17:46:46.894640 5726 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5faff593dc64d52a7634a9aa96d6298" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35357 } }
I20260928 17:46:46.894707 5726 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:46.894737 5726 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:46.894781 5726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:46.895278 5726 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5faff593dc64d52a7634a9aa96d6298" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35357 } }
I20260928 17:46:46.895367 5726 leader_election.cc:302] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5faff593dc64d52a7634a9aa96d6298; no voters:
I20260928 17:46:46.895509 5726 leader_election.cc:288] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:46.895545 5731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:46.895690 5731 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [term 1 LEADER]: Becoming Leader. State: Replica: b5faff593dc64d52a7634a9aa96d6298, State: Running, Role: LEADER
I20260928 17:46:46.895740 5726 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:46.895790 5731 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5faff593dc64d52a7634a9aa96d6298" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35357 } }
I20260928 17:46:46.896059 5731 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5faff593dc64d52a7634a9aa96d6298. Latest consensus state: current_term: 1 leader_uuid: "b5faff593dc64d52a7634a9aa96d6298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5faff593dc64d52a7634a9aa96d6298" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35357 } } }
I20260928 17:46:46.896122 5732 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5faff593dc64d52a7634a9aa96d6298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5faff593dc64d52a7634a9aa96d6298" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35357 } } }
I20260928 17:46:46.896237 5731 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:46.896278 5732 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:46.896683 5740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:46.897214 5740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 17:46:46 dist-test-slave-w3bm krb5kdc[5635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:46.898628 5740 catalog_manager.cc:1347] Generated new cluster ID: ee2510bb4abe4c1f88df7749d1327f2e
I20260928 17:46:46.898674 5740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:46:46.902380 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:46.892242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.3.222.62:35357)
0928 17:46:46.892373 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:46.892383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:46.892412 (+ 29us) server_negotiation.cc:407] Connection header received
0928 17:46:46.892841 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:46.892848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:46.892894 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:46.893021 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:46.893578 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:46.894417 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:46.895237 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:46.895742 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:46.895842 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:46.898776 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:46.898779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:46.898780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:46.898790 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:46.900551 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:46.900935 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:46.900937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:46.900938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:46.901027 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:46.901209 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:46.901210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:46.901210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:46.901462 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:46.902012 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:46.902084 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 17:46:46.902206 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":62,"threads_started":1}
I20260928 17:46:46.903035 5740 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:46.903523 5740 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:46:46.907552 5740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298: Generated new TSK 0
I20260928 17:46:46.907688 5740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:46:46.935797 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35357
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:47.031353 5752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:47.031543 5752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:47.031565 5752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:47.032655 5752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:47.032692 5752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:47.032711 5752 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 17:46:47.032724 5752 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:46:47.032754 5752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:46:47.034142 5752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.222.62:35357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:47.034346 5752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:47.034551 5752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:47.039116 5757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:47.039105 5760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:47.039098 5758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:47.039260 5752 server_base.cc:1034] running on GCE node
I20260928 17:46:47.039561 5752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:47.039788 5752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:47.040928 5752 hybrid_clock.cc:648] HybridClock initialized: now 1790617607040914 us; error 26 us; skew 500 ppm
Sep 28 17:46:47 dist-test-slave-w3bm krb5kdc[5635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:47.043104 5752 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:46:47.043514 5752 webserver.cc:460] Webserver started at http://127.3.222.1:33835/ using document root <none> and password file <none>
I20260928 17:46:47.043695 5752 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:47.043736 5752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:47.043855 5752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:47.044821 5752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "b0104f7b5ccb4bbab642e7df533bb2b8"
format_stamp: "Formatted at 2026-09-28 17:46:47 on dist-test-slave-w3bm"
server_key: "c522713ebd09a56ba1ee9dd667421e92"
server_key_iv: "795ecdaa50f19799be1b2b37157d5a66"
server_key_version: "encryptionkey@0"
I20260928 17:46:47.045126 5752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "b0104f7b5ccb4bbab642e7df533bb2b8"
format_stamp: "Formatted at 2026-09-28 17:46:47 on dist-test-slave-w3bm"
server_key: "c522713ebd09a56ba1ee9dd667421e92"
server_key_iv: "795ecdaa50f19799be1b2b37157d5a66"
server_key_version: "encryptionkey@0"
I20260928 17:46:47.046263 5752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:47.046865 5767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.047008 5752 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:47.047083 5752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "b0104f7b5ccb4bbab642e7df533bb2b8"
format_stamp: "Formatted at 2026-09-28 17:46:47 on dist-test-slave-w3bm"
server_key: "c522713ebd09a56ba1ee9dd667421e92"
server_key_iv: "795ecdaa50f19799be1b2b37157d5a66"
server_key_version: "encryptionkey@0"
I20260928 17:46:47.047142 5752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:47.055333 5752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:47.056862 5752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:47.056983 5752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:47.057413 5752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:47.057461 5752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.057487 5752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:47.057503 5752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 17:46:47 dist-test-slave-w3bm krb5kdc[5635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:47.064116 5752 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:46811
I20260928 17:46:47.064188 5882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:46811 every 8 connection(s)
I20260928 17:46:47.064457 5752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:46:47.066319 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.057833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:51751 (local address 127.3.222.62:35357)
0928 17:46:47.057891 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.057893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.058084 (+ 191us) server_negotiation.cc:407] Connection header received
0928 17:46:47.058548 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.058549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.058560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.058641 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:47.059476 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.060249 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.061320 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.061535 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.061589 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.063735 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:47.063737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:47.063737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:47.063744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:47.065136 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.065481 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.065482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.065483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.065539 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.065733 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.065734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.065734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.065842 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:47.065895 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.066139 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.066227 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:46:47.066452 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.057836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35357 (local address 127.3.222.1:51751)
0928 17:46:47.057978 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.058010 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.058460 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.058744 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.058747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.059053 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:47.059333 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.059340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.060337 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.060338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.061243 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.061246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.061610 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.061611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.061714 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.062171 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:47.062180 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:47.063654 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:47.065206 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.065211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.065213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.065419 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.065590 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.065591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.065591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.065681 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.065897 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:47.065908 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:47.066029 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.066113 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.066254 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260928 17:46:47.066884 5883 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35357
I20260928 17:46:47.066967 5883 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:47.067123 5883 heartbeater.cc:511] Master 127.3.222.62:35357 requested a full tablet report, sending...
I20260928 17:46:47.067610 5679 ts_manager.cc:194] Registered new tserver with Master: b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811)
I20260928 17:46:47.068252 5679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:51751
I20260928 17:46:47.070329 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 5752
I20260928 17:46:47.070425 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:46:47.070611 3960 external_mini_cluster.cc:1428] Setting key ef085b1497238f418bc4b7fc4d6834b8
I20260928 17:46:47.072743 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.067850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:37677 (local address 127.3.222.62:35357)
0928 17:46:47.067890 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.067891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.067979 (+ 88us) server_negotiation.cc:407] Connection header received
0928 17:46:47.068035 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.068035 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.068045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.068087 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:47.068352 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.068843 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.069509 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.069702 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.069752 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.070586 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:47.070587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:47.070588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:47.070593 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:47.071750 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.072074 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.072075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.072075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.072116 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.072253 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.072253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.072254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.072369 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:47.072426 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.072587 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.072656 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:46:47.072748 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.067797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35357 (local address 127.3.222.1:37677)
0928 17:46:47.067919 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.067929 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.067984 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.068090 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.068090 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.068245 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:47.068294 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.068296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.068923 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.068924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.069445 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.069447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.069757 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.069758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.069833 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.070170 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:47.070175 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:47.070515 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:47.071821 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.071822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.071822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.072014 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.072153 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.072154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.072154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.072204 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.072429 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:47.072430 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:47.072475 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.072562 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.072656 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:46:47.094231 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35357
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:47.171902 5892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:47.172092 5892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:47.172125 5892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:47.173270 5892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:47.173307 5892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:47.173334 5892 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 17:46:47.173349 5892 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:46:47.173372 5892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:46:47.174790 5892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.222.62:35357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:47.175019 5892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:47.175243 5892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:47.179946 5900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:47.179926 5898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:47.179919 5897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:47.179989 5892 server_base.cc:1034] running on GCE node
I20260928 17:46:47.180258 5892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:47.180464 5892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:47.181602 5892 hybrid_clock.cc:648] HybridClock initialized: now 1790617607181583 us; error 33 us; skew 500 ppm
Sep 28 17:46:47 dist-test-slave-w3bm krb5kdc[5635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:47.184051 5892 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:46:47.184444 5892 webserver.cc:460] Webserver started at http://127.3.222.2:41901/ using document root <none> and password file <none>
I20260928 17:46:47.184644 5892 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:47.184689 5892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:47.184808 5892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:47.185752 5892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "65e85c18831e4a648c28dc6d8cbcefe8"
format_stamp: "Formatted at 2026-09-28 17:46:47 on dist-test-slave-w3bm"
server_key: "e05799fba846a098fa5c3685b0194682"
server_key_iv: "12d961199d0122bd8478f422a5ee088a"
server_key_version: "encryptionkey@0"
I20260928 17:46:47.186057 5892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "65e85c18831e4a648c28dc6d8cbcefe8"
format_stamp: "Formatted at 2026-09-28 17:46:47 on dist-test-slave-w3bm"
server_key: "e05799fba846a098fa5c3685b0194682"
server_key_iv: "12d961199d0122bd8478f422a5ee088a"
server_key_version: "encryptionkey@0"
I20260928 17:46:47.187186 5892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:47.187899 5907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.188040 5892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:47.188114 5892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "65e85c18831e4a648c28dc6d8cbcefe8"
format_stamp: "Formatted at 2026-09-28 17:46:47 on dist-test-slave-w3bm"
server_key: "e05799fba846a098fa5c3685b0194682"
server_key_iv: "12d961199d0122bd8478f422a5ee088a"
server_key_version: "encryptionkey@0"
I20260928 17:46:47.188176 5892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:47.206856 5892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:47.208652 5892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:47.208791 5892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:47.209259 5892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:47.209298 5892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.209329 5892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:47.209347 5892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 17:46:47 dist-test-slave-w3bm krb5kdc[5635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:47.215782 5892 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:34141
I20260928 17:46:47.215852 6022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:34141 every 8 connection(s)
I20260928 17:46:47.216138 5892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:46:47.217820 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.209717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:52799 (local address 127.3.222.62:35357)
0928 17:46:47.209780 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.209783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.209943 (+ 160us) server_negotiation.cc:407] Connection header received
0928 17:46:47.210405 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.210407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.210420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.210485 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:47.211275 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.212032 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.213090 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.213285 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.213339 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.215470 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:47.215471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:47.215472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:47.215479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:47.216843 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.217165 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.217166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.217167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.217209 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.217364 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.217365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.217365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.217457 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:47.217499 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.217705 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.217759 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:46:47.217980 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.209677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35357 (local address 127.3.222.2:52799)
0928 17:46:47.209820 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.209860 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.210320 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.210573 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.210576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.210874 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:47.211159 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.211166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.212121 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.212122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.213013 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.213015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.213361 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.213362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.213491 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.214011 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:47.214021 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:47.215382 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:47.216909 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.216911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.216913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.217110 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.217244 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.217244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.217244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.217326 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.217503 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:47.217507 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:47.217615 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.217691 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.217800 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260928 17:46:47.218406 6023 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35357
I20260928 17:46:47.218490 6023 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:47.218533 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 5892
I20260928 17:46:47.218624 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:46:47.218693 6023 heartbeater.cc:511] Master 127.3.222.62:35357 requested a full tablet report, sending...
I20260928 17:46:47.218791 3960 external_mini_cluster.cc:1428] Setting key ca7db3d1826c8ab2d0761caf9a336ca8
I20260928 17:46:47.219055 5680 ts_manager.cc:194] Registered new tserver with Master: 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141)
I20260928 17:46:47.219534 5680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:52799
I20260928 17:46:47.224541 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.219374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35357 (local address 127.3.222.2:50817)
0928 17:46:47.219525 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.219536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.219582 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.219762 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.219762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.219942 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:47.220012 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.220014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.220723 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.220724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.221218 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.221219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.221599 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.221599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.221649 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.222014 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:47.222019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:47.222263 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:47.223716 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.223717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.223717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.223862 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.224035 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.224035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.224035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.224076 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.224339 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:47.224339 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:47.224370 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.224419 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.224482 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260928 17:46:47.224611 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.219369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:50817 (local address 127.3.222.62:35357)
0928 17:46:47.219663 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.219665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.219672 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:46:47.219689 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.219690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.219698 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.219747 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:47.220086 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.220633 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.221287 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.221537 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.221598 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.222342 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:47.222343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:47.222344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:47.222350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:47.223648 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.223924 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.223925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.223926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.223980 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.224112 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.224113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.224113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.224285 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:47.224350 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.224452 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.224529 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:46:47.242570 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35357
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:47.320659 6032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:47.320844 6032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:47.320866 6032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:47.321915 6032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:47.321954 6032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:47.321972 6032 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 17:46:47.321982 6032 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:46:47.321997 6032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:46:47.323364 6032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.222.62:35357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:47.323534 6032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:47.323694 6032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:47.328255 6037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:47.328224 6038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:47.328234 6032 server_base.cc:1034] running on GCE node
W20260928 17:46:47.328351 6040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:47.328670 6032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:47.328904 6032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:47.330039 6032 hybrid_clock.cc:648] HybridClock initialized: now 1790617607330027 us; error 27 us; skew 500 ppm
Sep 28 17:46:47 dist-test-slave-w3bm krb5kdc[5635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:47.332412 6032 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:46:47.332844 6032 webserver.cc:460] Webserver started at http://127.3.222.3:45709/ using document root <none> and password file <none>
I20260928 17:46:47.333086 6032 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:47.333135 6032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:47.333225 6032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:47.334151 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "a60d6ee16cc7428b9609ac9114addc7c"
format_stamp: "Formatted at 2026-09-28 17:46:47 on dist-test-slave-w3bm"
server_key: "009fb12a35a65e3a454f092f8cf6079c"
server_key_iv: "226d1725adea073e9a5be6b9d80ce2f1"
server_key_version: "encryptionkey@0"
I20260928 17:46:47.334420 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "a60d6ee16cc7428b9609ac9114addc7c"
format_stamp: "Formatted at 2026-09-28 17:46:47 on dist-test-slave-w3bm"
server_key: "009fb12a35a65e3a454f092f8cf6079c"
server_key_iv: "226d1725adea073e9a5be6b9d80ce2f1"
server_key_version: "encryptionkey@0"
I20260928 17:46:47.335543 6032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:46:47.336166 6047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.336369 6032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:46:47.336447 6032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "a60d6ee16cc7428b9609ac9114addc7c"
format_stamp: "Formatted at 2026-09-28 17:46:47 on dist-test-slave-w3bm"
server_key: "009fb12a35a65e3a454f092f8cf6079c"
server_key_iv: "226d1725adea073e9a5be6b9d80ce2f1"
server_key_version: "encryptionkey@0"
I20260928 17:46:47.336498 6032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:47.351505 6032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:47.353256 6032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:47.353358 6032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:47.353819 6032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:47.353854 6032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.353875 6032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:47.353888 6032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 17:46:47 dist-test-slave-w3bm krb5kdc[5635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:47.360543 6032 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:36929
I20260928 17:46:47.360599 6162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:36929 every 8 connection(s)
I20260928 17:46:47.360913 6032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:46:47.362794 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.354213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:49375 (local address 127.3.222.62:35357)
0928 17:46:47.354269 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.354271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.354498 (+ 227us) server_negotiation.cc:407] Connection header received
0928 17:46:47.355125 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.355127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.355138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.355178 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:47.356005 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.356589 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.357614 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.357820 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.357874 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.360461 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:47.360463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:47.360463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:47.360470 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:47.361813 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.362145 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.362146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.362147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.362195 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.362343 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.362344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.362344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.362433 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:47.362478 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.362679 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.362734 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:46:47.362962 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.354195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35357 (local address 127.3.222.3:49375)
0928 17:46:47.354364 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.354409 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.355032 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.355277 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.355282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.355622 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:47.355912 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.355919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.356681 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.356682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.357537 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.357540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.357889 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.357891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.358019 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.358449 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:47.358459 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:47.360374 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:47.361887 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.361892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.361893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.362088 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.362234 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.362235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.362235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.362311 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.362489 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:47.362491 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:47.362594 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.362665 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.362772 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260928 17:46:47.363404 6163 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35357
I20260928 17:46:47.363481 6163 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:47.363649 6163 heartbeater.cc:511] Master 127.3.222.62:35357 requested a full tablet report, sending...
I20260928 17:46:47.363998 5679 ts_manager.cc:194] Registered new tserver with Master: a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929)
I20260928 17:46:47.364368 5679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:49375
I20260928 17:46:47.366778 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 6032
I20260928 17:46:47.366873 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:46:47.367079 3960 external_mini_cluster.cc:1428] Setting key 2ab59b001f8c74106f652305a6dc2db6
I20260928 17:46:47.367884 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:46:47.369199 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.364360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:53147 (local address 127.3.222.62:35357)
0928 17:46:47.364397 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.364399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.364474 (+ 75us) server_negotiation.cc:407] Connection header received
0928 17:46:47.364557 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.364558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.364567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.364622 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:47.364915 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.365351 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.365994 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.366182 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.366232 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.367032 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:47.367033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:47.367033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:47.367038 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:47.368361 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.368676 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.368676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.368677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.368716 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.368851 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.368852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.368852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.368933 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:47.368976 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.369097 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.369144 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:46:47.369251 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.364297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35357 (local address 127.3.222.3:53147)
0928 17:46:47.364407 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.364418 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.364499 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.364623 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.364624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.364800 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:47.364855 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.364857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.365422 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.365423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.365919 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.365921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.366234 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.366235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.366282 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.366608 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:47.366612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:47.366960 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:47.368427 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.368428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.368428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.368618 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.368753 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.368754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.368754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.368807 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.368980 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:47.368981 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:47.369018 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.369082 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.369167 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
Sep 28 17:46:47 dist-test-slave-w3bm krb5kdc[5635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:46:47 dist-test-slave-w3bm krb5kdc[5635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:47.388265 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.382333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.3.222.62:35357)
0928 17:46:47.382378 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.382380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.382389 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:46:47.382415 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.382416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.382424 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.382479 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:47.382984 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.383526 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.384202 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.384379 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.384427 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.386374 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:47.386375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:47.386376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:47.386382 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:47.387460 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.387734 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.387735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.387735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.387781 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.387905 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.387905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.387905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.388016 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:47.388082 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.388150 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.388201 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:46:47.389693 5679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:46:47.391264 5679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:47.397140 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.393279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.3.222.2:34141)
0928 17:46:47.393414 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.393419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.393544 (+ 125us) server_negotiation.cc:407] Connection header received
0928 17:46:47.393622 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.393624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.393639 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.393697 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:47.393818 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.394654 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.395898 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.396602 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.396770 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.396939 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.397014 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.397050 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260928 17:46:47.397248 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.393103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:34141 (local address 127.0.0.1:41612)
0928 17:46:47.393428 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.393483 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.393541 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.393703 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.393705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.393707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:47.393766 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.393768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.394722 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.394723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.395772 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.395775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.396781 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.396782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.396855 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.396913 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.397122 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.397180 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":57,"threads_started":1}
I20260928 17:46:47.397337 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.393113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:46811 (local address 127.0.0.1:55348)
0928 17:46:47.393793 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.393802 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.393849 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.394051 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.394052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.394054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:47.394085 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.394087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.395278 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.395278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.396181 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.396183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.397123 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.397124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.397183 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.397188 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.397232 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.397271 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":57,"threads_started":1}
I20260928 17:46:47.397518 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.393104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:36929 (local address 127.0.0.1:41580)
0928 17:46:47.393431 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.393484 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.393538 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.393730 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.393730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.393732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:47.393783 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.393785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.395205 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.395227 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.396363 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.396366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.397313 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.397314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.397368 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.397371 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.397415 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.397448 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":23,"thread_start_us":54,"threads_started":1}
I20260928 17:46:47.397629 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.393337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.3.222.1:46811)
0928 17:46:47.393604 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.393607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.393854 (+ 247us) server_negotiation.cc:407] Connection header received
0928 17:46:47.393902 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.393905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.393920 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.393989 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:47.394146 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.395202 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.396249 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.396973 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.397038 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.397091 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.397501 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.397537 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":67,"threads_started":1}
I20260928 17:46:47.397905 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.393263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.3.222.3:36929)
0928 17:46:47.393505 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.393510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.393533 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:46:47.393641 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.393643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.393658 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.393723 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:47.393978 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.395123 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.396436 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.397183 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.397239 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.397284 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.397659 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.397819 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260928 17:46:47.398082 5957 tablet_service.cc:1467] Processing CreateTablet for tablet 38f4680582b94a46b367406ac2036548 (DEFAULT_TABLE table=test-table [id=e9253e471f49473ba95952e5c897eac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:47.398273 5817 tablet_service.cc:1467] Processing CreateTablet for tablet 38f4680582b94a46b367406ac2036548 (DEFAULT_TABLE table=test-table [id=e9253e471f49473ba95952e5c897eac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:47.398414 5957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f4680582b94a46b367406ac2036548. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:47.398567 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f4680582b94a46b367406ac2036548. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:47.398535 6097 tablet_service.cc:1467] Processing CreateTablet for tablet 38f4680582b94a46b367406ac2036548 (DEFAULT_TABLE table=test-table [id=e9253e471f49473ba95952e5c897eac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:47.398787 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f4680582b94a46b367406ac2036548. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:47.401254 6183 tablet_bootstrap.cc:492] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: Bootstrap starting.
I20260928 17:46:47.401638 6184 tablet_bootstrap.cc:492] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: Bootstrap starting.
I20260928 17:46:47.401790 6183 tablet_bootstrap.cc:654] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:47.402040 6185 tablet_bootstrap.cc:492] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: Bootstrap starting.
I20260928 17:46:47.402334 6183 log.cc:824] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:47.402419 6184 tablet_bootstrap.cc:654] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:47.402630 6185 tablet_bootstrap.cc:654] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:47.402742 6184 log.cc:824] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:47.402937 6185 log.cc:824] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:47.403057 6183 tablet_bootstrap.cc:492] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: No bootstrap required, opened a new log
I20260928 17:46:47.403122 6183 ts_tablet_manager.cc:1397] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:47.403411 6184 tablet_bootstrap.cc:492] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: No bootstrap required, opened a new log
I20260928 17:46:47.403479 6184 ts_tablet_manager.cc:1397] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:47.403672 6185 tablet_bootstrap.cc:492] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: No bootstrap required, opened a new log
I20260928 17:46:47.403761 6185 ts_tablet_manager.cc:1397] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:47.404619 6183 raft_consensus.cc:348] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } }
I20260928 17:46:47.404742 6183 raft_consensus.cc:374] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:47.404772 6183 raft_consensus.cc:729] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0104f7b5ccb4bbab642e7df533bb2b8, State: Initialized, Role: FOLLOWER
I20260928 17:46:47.404912 6183 consensus_queue.cc:260] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } }
I20260928 17:46:47.405146 6183 ts_tablet_manager.cc:1428] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:46:47.405259 5883 heartbeater.cc:503] Master 127.3.222.62:35357 was elected leader, sending a full tablet report...
I20260928 17:46:47.405340 6184 raft_consensus.cc:348] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } }
I20260928 17:46:47.405539 6184 raft_consensus.cc:374] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:47.405596 6184 raft_consensus.cc:729] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60d6ee16cc7428b9609ac9114addc7c, State: Initialized, Role: FOLLOWER
I20260928 17:46:47.405555 6185 raft_consensus.cc:348] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } }
I20260928 17:46:47.405687 6185 raft_consensus.cc:374] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:47.405725 6185 raft_consensus.cc:729] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e85c18831e4a648c28dc6d8cbcefe8, State: Initialized, Role: FOLLOWER
I20260928 17:46:47.405707 6184 consensus_queue.cc:260] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } }
I20260928 17:46:47.405822 6185 consensus_queue.cc:260] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } }
I20260928 17:46:47.405931 6184 ts_tablet_manager.cc:1428] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:46:47.406062 6163 heartbeater.cc:503] Master 127.3.222.62:35357 was elected leader, sending a full tablet report...
I20260928 17:46:47.406071 6185 ts_tablet_manager.cc:1428] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:46:47.406247 6023 heartbeater.cc:503] Master 127.3.222.62:35357 was elected leader, sending a full tablet report...
W20260928 17:46:47.466658 6024 tablet.cc:2369] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:46:47.565138 5884 tablet.cc:2369] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:46:47.577314 6191 raft_consensus.cc:482] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:47.577421 6191 raft_consensus.cc:504] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } }
I20260928 17:46:47.577718 6191 leader_election.cc:288] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929), b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811)
I20260928 17:46:47.580323 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.577968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:57037 (local address 127.3.222.3:36929)
0928 17:46:47.578021 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.578023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.578054 (+ 31us) server_negotiation.cc:407] Connection header received
0928 17:46:47.578077 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.578078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.578087 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.578129 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:47.578264 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.578742 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.579711 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.580047 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.580110 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.580121 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.580227 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.580261 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 17:46:47.580358 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.577923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:43109 (local address 127.3.222.1:46811)
0928 17:46:47.577967 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.577969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.578127 (+ 158us) server_negotiation.cc:407] Connection header received
0928 17:46:47.578161 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.578162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.578173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.578218 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:47.578344 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.578789 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.579719 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.580073 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.580129 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.580138 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.580259 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.580293 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:46:47.580374 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.577882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:36929 (local address 127.3.222.2:57037)
0928 17:46:47.577959 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.577970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.578015 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.578143 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.578144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.578146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:47.578205 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.578207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.578807 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.578808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.579637 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.579639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.580109 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.580110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.580172 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.580175 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.580217 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.580255 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":50}
I20260928 17:46:47.580381 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.577885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:46811 (local address 127.3.222.2:43109)
0928 17:46:47.578062 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.578071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.578118 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.578223 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.578223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.578225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:47.578289 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.578290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.578853 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.578854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.579640 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.579642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.580137 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.580138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.580203 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.580206 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.580249 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.580284 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":13,"thread_start_us":111,"threads_started":1}
I20260928 17:46:47.580760 5837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f4680582b94a46b367406ac2036548" candidate_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" is_pre_election: true
I20260928 17:46:47.580909 5837 raft_consensus.cc:2454] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e85c18831e4a648c28dc6d8cbcefe8 in term 0.
I20260928 17:46:47.580960 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f4680582b94a46b367406ac2036548" candidate_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60d6ee16cc7428b9609ac9114addc7c" is_pre_election: true
I20260928 17:46:47.581115 6117 raft_consensus.cc:2454] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e85c18831e4a648c28dc6d8cbcefe8 in term 0.
I20260928 17:46:47.581148 5909 leader_election.cc:302] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e85c18831e4a648c28dc6d8cbcefe8, b0104f7b5ccb4bbab642e7df533bb2b8; no voters:
I20260928 17:46:47.581287 6191 raft_consensus.cc:2781] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:47.581336 6191 raft_consensus.cc:482] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:47.581357 6191 raft_consensus.cc:3037] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:47.582249 6191 raft_consensus.cc:504] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } }
I20260928 17:46:47.582420 6191 leader_election.cc:288] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [CANDIDATE]: Term 1 election: Requested vote from peers a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929), b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811)
I20260928 17:46:47.582487 5837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f4680582b94a46b367406ac2036548" candidate_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8"
I20260928 17:46:47.582510 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f4680582b94a46b367406ac2036548" candidate_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60d6ee16cc7428b9609ac9114addc7c"
I20260928 17:46:47.582566 6117 raft_consensus.cc:3037] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:47.582566 5837 raft_consensus.cc:3037] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:47.583425 5837 raft_consensus.cc:2454] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e85c18831e4a648c28dc6d8cbcefe8 in term 1.
I20260928 17:46:47.583425 6117 raft_consensus.cc:2454] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e85c18831e4a648c28dc6d8cbcefe8 in term 1.
I20260928 17:46:47.583563 5909 leader_election.cc:302] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e85c18831e4a648c28dc6d8cbcefe8, b0104f7b5ccb4bbab642e7df533bb2b8; no voters:
I20260928 17:46:47.583638 6191 raft_consensus.cc:2781] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:47.583778 6191 raft_consensus.cc:686] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 LEADER]: Becoming Leader. State: Replica: 65e85c18831e4a648c28dc6d8cbcefe8, State: Running, Role: LEADER
I20260928 17:46:47.583853 6191 consensus_queue.cc:237] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } }
I20260928 17:46:47.584591 5680 catalog_manager.cc:5495] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2). New cstate: current_term: 1 leader_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } health_report { overall_health: UNKNOWN } } }
W20260928 17:46:47.612183 6164 tablet.cc:2369] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 17:46:47 dist-test-slave-w3bm krb5kdc[5635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:47.641830 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.635859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.3.222.62:35357)
0928 17:46:47.635911 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.635914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.636022 (+ 108us) server_negotiation.cc:407] Connection header received
0928 17:46:47.636073 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.636074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.636082 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.636127 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:47.636602 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.637081 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.637703 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.637898 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.637942 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.639911 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:47.639912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:47.639913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:47.639918 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:47.640967 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.641269 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.641270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.641270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.641336 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:47.641464 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:47.641465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:47.641465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:47.641559 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:47.641601 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.641699 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.641760 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:46:47.641934 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.635803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35357 (local address 127.0.0.1:55756)
0928 17:46:47.635958 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.635967 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.636033 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.636136 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.636137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.636480 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:47.636546 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.636548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.637134 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.637134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.637654 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.637656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.637943 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.637944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.637988 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.638458 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:47.638463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:47.639824 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:47.641031 (+ 1207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.641037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.641039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.641217 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.641381 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:47.641381 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:47.641381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:47.641426 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:47.641598 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:47.641601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:47.641638 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.641683 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.641825 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":61,"thread_start_us":82,"threads_started":1}
I20260928 17:46:47.643855 5680 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:55756:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 17:46:47.644054 5680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:47.645367 5817 tablet_service.cc:1467] Processing CreateTablet for tablet 507de8e83b1949b0bb8bbf73dd732c73 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87d9e8a53fab48eea7c4715c47d16e6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:46:47.645388 6097 tablet_service.cc:1467] Processing CreateTablet for tablet 507de8e83b1949b0bb8bbf73dd732c73 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87d9e8a53fab48eea7c4715c47d16e6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:46:47.645525 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507de8e83b1949b0bb8bbf73dd732c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:47.645525 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507de8e83b1949b0bb8bbf73dd732c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:47.646581 6183 tablet_bootstrap.cc:492] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: Bootstrap starting.
I20260928 17:46:47.646584 6184 tablet_bootstrap.cc:492] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: Bootstrap starting.
I20260928 17:46:47.646677 5957 tablet_service.cc:1467] Processing CreateTablet for tablet 507de8e83b1949b0bb8bbf73dd732c73 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87d9e8a53fab48eea7c4715c47d16e6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:46:47.646817 5957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507de8e83b1949b0bb8bbf73dd732c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:47.646947 6183 tablet_bootstrap.cc:654] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:47.647092 6184 tablet_bootstrap.cc:654] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:47.647526 6183 tablet_bootstrap.cc:492] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: No bootstrap required, opened a new log
I20260928 17:46:47.647576 6183 ts_tablet_manager.cc:1397] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:46:47.647732 6183 raft_consensus.cc:348] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.647805 6183 raft_consensus.cc:374] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:47.647825 6183 raft_consensus.cc:729] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0104f7b5ccb4bbab642e7df533bb2b8, State: Initialized, Role: FOLLOWER
I20260928 17:46:47.647866 6183 consensus_queue.cc:260] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.647897 6184 tablet_bootstrap.cc:492] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: No bootstrap required, opened a new log
I20260928 17:46:47.647944 6184 ts_tablet_manager.cc:1397] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:46:47.647962 6183 ts_tablet_manager.cc:1428] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 17:46:47.648090 6189 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.648083 6184 raft_consensus.cc:348] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.648145 6189 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:47.648154 6184 raft_consensus.cc:374] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:47.648172 6184 raft_consensus.cc:729] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60d6ee16cc7428b9609ac9114addc7c, State: Initialized, Role: FOLLOWER
I20260928 17:46:47.648213 6184 consensus_queue.cc:260] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.648280 6185 tablet_bootstrap.cc:492] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: Bootstrap starting.
I20260928 17:46:47.648288 6184 ts_tablet_manager.cc:1428] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.648406 6190 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.648494 6190 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:47.648634 6185 tablet_bootstrap.cc:654] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:47.649259 6185 tablet_bootstrap.cc:492] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: No bootstrap required, opened a new log
I20260928 17:46:47.649300 6185 ts_tablet_manager.cc:1397] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:47.649426 6185 raft_consensus.cc:348] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.649484 6185 raft_consensus.cc:374] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:47.649501 6185 raft_consensus.cc:729] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e85c18831e4a648c28dc6d8cbcefe8, State: Initialized, Role: FOLLOWER
I20260928 17:46:47.649539 6185 consensus_queue.cc:260] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.649628 6185 ts_tablet_manager.cc:1428] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.649668 6191 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.649744 6191 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:47.671919 6190 raft_consensus.cc:482] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:47.672008 6190 raft_consensus.cc:504] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.672353 6190 leader_election.cc:288] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811), 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141)
I20260928 17:46:47.675015 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.672490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:34141 (local address 127.3.222.3:53289)
0928 17:46:47.672540 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.672552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.672599 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.672741 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.672742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.672744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:47.672814 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.672816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.673475 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.673476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.674244 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.674246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.674746 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.674747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.674839 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.674842 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.674887 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.674924 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":29}
I20260928 17:46:47.675015 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.672588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:49983 (local address 127.3.222.1:46811)
0928 17:46:47.672630 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.672632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.672722 (+ 90us) server_negotiation.cc:407] Connection header received
0928 17:46:47.672771 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.672771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.672780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.672838 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:47.672961 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.673416 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.674330 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.674682 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.674739 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.674750 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.674894 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.674945 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:46:47.675096 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.672572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:53289 (local address 127.3.222.2:34141)
0928 17:46:47.672633 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.672635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.672644 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:46:47.672670 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.672671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.672682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.672732 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:47.672897 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.673405 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.674325 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.674684 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.674741 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.674750 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.674977 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.675030 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:46:47.675222 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.672505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:46811 (local address 127.3.222.3:49983)
0928 17:46:47.672659 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.672668 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.672724 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.672839 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.672840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.672841 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:47.672905 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.672907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.673542 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.673543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.674251 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.674253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.674763 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.674764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.674830 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.674834 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.674886 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.674926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260928 17:46:47.675365 5977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507de8e83b1949b0bb8bbf73dd732c73" candidate_uuid: "a60d6ee16cc7428b9609ac9114addc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" is_pre_election: true
I20260928 17:46:47.675470 5977 raft_consensus.cc:2454] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60d6ee16cc7428b9609ac9114addc7c in term 0.
I20260928 17:46:47.675638 5837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507de8e83b1949b0bb8bbf73dd732c73" candidate_uuid: "a60d6ee16cc7428b9609ac9114addc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" is_pre_election: true
I20260928 17:46:47.675662 6048 leader_election.cc:302] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e85c18831e4a648c28dc6d8cbcefe8, a60d6ee16cc7428b9609ac9114addc7c; no voters:
I20260928 17:46:47.675735 5837 raft_consensus.cc:2454] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60d6ee16cc7428b9609ac9114addc7c in term 0.
I20260928 17:46:47.675787 6190 raft_consensus.cc:2781] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:47.675835 6190 raft_consensus.cc:482] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:47.675855 6190 raft_consensus.cc:3037] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:47.676512 6190 raft_consensus.cc:504] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.676617 6190 leader_election.cc:288] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [CANDIDATE]: Term 1 election: Requested vote from peers b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811), 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141)
I20260928 17:46:47.676764 5837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507de8e83b1949b0bb8bbf73dd732c73" candidate_uuid: "a60d6ee16cc7428b9609ac9114addc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8"
I20260928 17:46:47.676836 5837 raft_consensus.cc:3037] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:47.677408 5837 raft_consensus.cc:2454] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60d6ee16cc7428b9609ac9114addc7c in term 1.
I20260928 17:46:47.677510 5977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507de8e83b1949b0bb8bbf73dd732c73" candidate_uuid: "a60d6ee16cc7428b9609ac9114addc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e85c18831e4a648c28dc6d8cbcefe8"
I20260928 17:46:47.677556 6049 leader_election.cc:302] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60d6ee16cc7428b9609ac9114addc7c, b0104f7b5ccb4bbab642e7df533bb2b8; no voters:
I20260928 17:46:47.677585 5977 raft_consensus.cc:3037] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:47.677639 6190 raft_consensus.cc:2781] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:47.677770 6190 raft_consensus.cc:686] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [term 1 LEADER]: Becoming Leader. State: Replica: a60d6ee16cc7428b9609ac9114addc7c, State: Running, Role: LEADER
I20260928 17:46:47.677863 6190 consensus_queue.cc:237] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.678336 5977 raft_consensus.cc:2454] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60d6ee16cc7428b9609ac9114addc7c in term 1.
I20260928 17:46:47.678544 6205 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: TxnStatusTablet state changed. Reason: New leader a60d6ee16cc7428b9609ac9114addc7c. Latest consensus state: current_term: 1 leader_uuid: "a60d6ee16cc7428b9609ac9114addc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.678624 6205 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:46:47.678668 5680 catalog_manager.cc:5495] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3). New cstate: current_term: 1 leader_uuid: "a60d6ee16cc7428b9609ac9114addc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } health_report { overall_health: HEALTHY } } }
I20260928 17:46:47.678793 6207 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 17:46:47.678901 6207 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 17:46:47.687412 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.684850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.3.222.3:36929)
0928 17:46:47.684895 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.684896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.684902 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:46:47.684926 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.684926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.684941 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.684993 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:47.685137 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.685617 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.686668 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.686860 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.686918 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.687097 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.687189 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.687272 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.687342 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.687362 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:46:47.687453 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.684635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:36929 (local address 127.0.0.1:41584)
0928 17:46:47.684707 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.684719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.684766 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.685028 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.685029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.685030 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:46:47.685079 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.685081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.685685 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.685686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.686578 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.686581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.686930 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.686931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.687005 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.687022 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.687265 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.687272 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.687327 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.687374 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 17:46:47.689124 5837 raft_consensus.cc:1261] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Refusing update from remote peer a60d6ee16cc7428b9609ac9114addc7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:47.689136 5977 raft_consensus.cc:1261] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Refusing update from remote peer a60d6ee16cc7428b9609ac9114addc7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:47.689395 6205 consensus_queue.cc:1035] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:47.689505 6190 consensus_queue.cc:1035] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:47.690634 6191 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: TxnStatusTablet state changed. Reason: New leader a60d6ee16cc7428b9609ac9114addc7c. Latest consensus state: current_term: 1 leader_uuid: "a60d6ee16cc7428b9609ac9114addc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.690721 6191 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:47.690791 6189 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: TxnStatusTablet state changed. Reason: New leader a60d6ee16cc7428b9609ac9114addc7c. Latest consensus state: current_term: 1 leader_uuid: "a60d6ee16cc7428b9609ac9114addc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.690857 6189 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:47.691574 6205 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a60d6ee16cc7428b9609ac9114addc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.691581 6190 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a60d6ee16cc7428b9609ac9114addc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.691656 6205 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:46:47.691674 6190 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P a60d6ee16cc7428b9609ac9114addc7c: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:46:47.691752 6189 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a60d6ee16cc7428b9609ac9114addc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.691829 6189 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P b0104f7b5ccb4bbab642e7df533bb2b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:47.692035 6191 tablet_replica.cc:439] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a60d6ee16cc7428b9609ac9114addc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } } }
I20260928 17:46:47.692065 6211 mvcc.cc:205] Tried to move back new op lower bound from 7334369721092136960 to 7334369721049083904. Current Snapshot: MvccSnapshot[applied={T|T < 7334369721092136960}]
I20260928 17:46:47.692081 6210 mvcc.cc:205] Tried to move back new op lower bound from 7334369721092136960 to 7334369721049083904. Current Snapshot: MvccSnapshot[applied={T|T < 7334369721092136960}]
I20260928 17:46:47.692109 6191 tablet_replica.cc:442] T 507de8e83b1949b0bb8bbf73dd732c73 P 65e85c18831e4a648c28dc6d8cbcefe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:46:47.693549 6212 mvcc.cc:205] Tried to move back new op lower bound from 7334369721092136960 to 7334369721049083904. Current Snapshot: MvccSnapshot[applied={T|T < 7334369721092136960}]
I20260928 17:46:47.696146 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.693764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.3.222.2:34141)
0928 17:46:47.693799 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.693801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.693827 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:46:47.693908 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.693909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.693922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.693980 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:47.694119 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.694579 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.695339 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.695640 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.695706 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.695842 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.695957 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.696042 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.696059 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.696081 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:46:47.700491 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.698174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:34607 (local address 127.3.222.3:36929)
0928 17:46:47.698208 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.698210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.698233 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:46:47.698299 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.698300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.698307 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.698344 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:47.698453 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.698875 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.699928 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.700098 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.700142 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.700318 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.700342 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.700386 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.700432 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.700448 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:46:47.700534 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.698127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:36929 (local address 127.3.222.2:34607)
0928 17:46:47.698185 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.698196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.698247 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.698347 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.698348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.698349 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:46:47.698401 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.698403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.698934 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.698935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.699843 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.699846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.700155 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.700156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.700219 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.700232 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.700386 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.700388 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.700430 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.700467 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:46:47.702561 5837 raft_consensus.cc:1261] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Refusing update from remote peer 65e85c18831e4a648c28dc6d8cbcefe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:47.702688 6117 raft_consensus.cc:1261] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Refusing update from remote peer 65e85c18831e4a648c28dc6d8cbcefe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:47.702767 6191 consensus_queue.cc:1035] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:47.702905 6193 consensus_queue.cc:1035] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:47.710685 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.708191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:38565 (local address 127.3.222.2:34141)
0928 17:46:47.708224 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.708226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.708237 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:46:47.708294 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.708294 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.708304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.708354 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:47.708495 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.708934 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.710004 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.710275 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.710329 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.710510 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.710536 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.710573 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.710626 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.710640 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:46:47.710729 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.708154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:34141 (local address 127.3.222.3:38565)
0928 17:46:47.708203 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.708213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.708250 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.708364 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.708365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.708366 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:46:47.708423 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.708424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.708991 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.708999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.709905 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.709908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.710359 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.710360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.710424 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.710437 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.710576 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.710578 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.710622 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.710659 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 17:46:47.720332 5680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:47.720449 5680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:47.721196 5680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298: Sending DeleteTablet for 3 replicas of tablet 38f4680582b94a46b367406ac2036548
I20260928 17:46:47.721681 5817 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f4680582b94a46b367406ac2036548 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:47 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:55348
I20260928 17:46:47.721716 5957 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f4680582b94a46b367406ac2036548 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:47 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:41612
I20260928 17:46:47.721892 6097 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f4680582b94a46b367406ac2036548 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:47 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:41580
I20260928 17:46:47.722088 6230 tablet_replica.cc:330] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: stopping tablet replica
I20260928 17:46:47.722076 5680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:46:47.722210 5680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:47.722220 6230 raft_consensus.cc:2229] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:47.722270 6230 raft_consensus.cc:2258] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:47.722685 6230 ts_tablet_manager.cc:1905] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:47.723865 5817 tablet_service.cc:1467] Processing CreateTablet for tablet a5c99c6ecf3243ffbe5f0fb036179a17 (DEFAULT_TABLE table=test-table [id=cf27361833db4c519916c8f85a6918a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:47.723912 5957 tablet_service.cc:1467] Processing CreateTablet for tablet a5c99c6ecf3243ffbe5f0fb036179a17 (DEFAULT_TABLE table=test-table [id=cf27361833db4c519916c8f85a6918a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:47.723974 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c99c6ecf3243ffbe5f0fb036179a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:47.724022 5957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c99c6ecf3243ffbe5f0fb036179a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:47.725244 6185 tablet_bootstrap.cc:492] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: Bootstrap starting.
I20260928 17:46:47.725244 6183 tablet_bootstrap.cc:492] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: Bootstrap starting.
I20260928 17:46:47.725620 6183 tablet_bootstrap.cc:654] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:47.725641 6185 tablet_bootstrap.cc:654] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:47.726260 6097 tablet_service.cc:1467] Processing CreateTablet for tablet a5c99c6ecf3243ffbe5f0fb036179a17 (DEFAULT_TABLE table=test-table [id=cf27361833db4c519916c8f85a6918a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:47.726359 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c99c6ecf3243ffbe5f0fb036179a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:47.726686 6230 ts_tablet_manager.cc:1918] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:46:47.726745 6230 log.cc:1196] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/38f4680582b94a46b367406ac2036548
I20260928 17:46:47.726969 6230 ts_tablet_manager.cc:1939] T 38f4680582b94a46b367406ac2036548 P a60d6ee16cc7428b9609ac9114addc7c: Deleting consensus metadata
I20260928 17:46:47.727567 6184 tablet_bootstrap.cc:492] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: Bootstrap starting.
I20260928 17:46:47.727685 5666 catalog_manager.cc:4841] TS a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929): tablet 38f4680582b94a46b367406ac2036548 (table test-table [id=e9253e471f49473ba95952e5c897eac8]) successfully deleted
I20260928 17:46:47.727917 6184 tablet_bootstrap.cc:654] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:47.728389 6185 tablet_bootstrap.cc:492] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: No bootstrap required, opened a new log
I20260928 17:46:47.728436 6185 ts_tablet_manager.cc:1397] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 17:46:47.728446 6184 tablet_bootstrap.cc:492] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: No bootstrap required, opened a new log
I20260928 17:46:47.728482 6184 ts_tablet_manager.cc:1397] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:46:47.728560 6185 raft_consensus.cc:348] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.728619 6185 raft_consensus.cc:374] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:47.728598 6184 raft_consensus.cc:348] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.728638 6185 raft_consensus.cc:729] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e85c18831e4a648c28dc6d8cbcefe8, State: Initialized, Role: FOLLOWER
I20260928 17:46:47.728654 6184 raft_consensus.cc:374] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:47.728672 6184 raft_consensus.cc:729] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60d6ee16cc7428b9609ac9114addc7c, State: Initialized, Role: FOLLOWER
I20260928 17:46:47.728677 6185 consensus_queue.cc:260] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.728711 6184 consensus_queue.cc:260] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.728762 6185 ts_tablet_manager.cc:1428] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.728785 6184 ts_tablet_manager.cc:1428] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.729095 6229 tablet_replica.cc:330] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: stopping tablet replica
I20260928 17:46:47.729233 6229 raft_consensus.cc:2229] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:46:47.729673 6229 raft_consensus.cc:2258] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:47.730116 6183 tablet_bootstrap.cc:492] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: No bootstrap required, opened a new log
I20260928 17:46:47.730165 6183 ts_tablet_manager.cc:1397] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 17:46:47.730266 6229 ts_tablet_manager.cc:1905] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:47.730293 6183 raft_consensus.cc:348] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.730361 6183 raft_consensus.cc:374] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:47.730381 6183 raft_consensus.cc:729] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0104f7b5ccb4bbab642e7df533bb2b8, State: Initialized, Role: FOLLOWER
I20260928 17:46:47.730427 6183 consensus_queue.cc:260] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.730504 6183 ts_tablet_manager.cc:1428] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:47.731990 6229 ts_tablet_manager.cc:1918] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:46:47.732059 6229 log.cc:1196] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/38f4680582b94a46b367406ac2036548
I20260928 17:46:47.732326 6229 ts_tablet_manager.cc:1939] T 38f4680582b94a46b367406ac2036548 P 65e85c18831e4a648c28dc6d8cbcefe8: Deleting consensus metadata
I20260928 17:46:47.732507 6228 tablet_replica.cc:330] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: stopping tablet replica
I20260928 17:46:47.732620 6228 raft_consensus.cc:2229] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:47.732734 6228 raft_consensus.cc:2258] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:47.732860 5664 catalog_manager.cc:4841] TS 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141): tablet 38f4680582b94a46b367406ac2036548 (table test-table [id=e9253e471f49473ba95952e5c897eac8]) successfully deleted
I20260928 17:46:47.733234 6228 ts_tablet_manager.cc:1905] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:47.734684 6228 ts_tablet_manager.cc:1918] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:46:47.734752 6228 log.cc:1196] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/38f4680582b94a46b367406ac2036548
I20260928 17:46:47.735011 6228 ts_tablet_manager.cc:1939] T 38f4680582b94a46b367406ac2036548 P b0104f7b5ccb4bbab642e7df533bb2b8: Deleting consensus metadata
I20260928 17:46:47.735397 5665 catalog_manager.cc:4841] TS b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811): tablet 38f4680582b94a46b367406ac2036548 (table test-table [id=e9253e471f49473ba95952e5c897eac8]) successfully deleted
I20260928 17:46:47.813549 6189 raft_consensus.cc:482] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:47.813666 6189 raft_consensus.cc:504] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.813997 6189 leader_election.cc:288] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141), a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929)
I20260928 17:46:47.817008 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.814361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:51947 (local address 127.3.222.2:34141)
0928 17:46:47.814421 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.814424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.814515 (+ 91us) server_negotiation.cc:407] Connection header received
0928 17:46:47.814617 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.814618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.814627 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.814682 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:47.814797 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.815312 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.816354 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.816699 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.816757 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.816767 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.816903 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.816941 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:46:47.817044 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.814174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:34141 (local address 127.3.222.1:51947)
0928 17:46:47.814445 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.814456 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.814540 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.814695 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.814695 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.814697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:47.814745 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.814746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.815382 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.815383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.816266 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.816269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.816767 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.816769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.816842 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.816846 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.816896 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.816937 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":28,"thread_start_us":62,"threads_started":1}
I20260928 17:46:47.817288 5977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c99c6ecf3243ffbe5f0fb036179a17" candidate_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" is_pre_election: true
I20260928 17:46:47.817373 5977 raft_consensus.cc:2454] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0104f7b5ccb4bbab642e7df533bb2b8 in term 0.
I20260928 17:46:47.817540 5768 leader_election.cc:302] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e85c18831e4a648c28dc6d8cbcefe8, b0104f7b5ccb4bbab642e7df533bb2b8; no voters:
I20260928 17:46:47.817663 6189 raft_consensus.cc:2781] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:47.817689 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.814242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:51029 (local address 127.3.222.3:36929)
0928 17:46:47.814298 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.814300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.814422 (+ 122us) server_negotiation.cc:407] Connection header received
0928 17:46:47.814529 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.814530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.814540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.814589 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:47.814762 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.815511 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.816840 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.817345 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.817402 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.817413 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.817599 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.817634 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:46:47.817714 6189 raft_consensus.cc:482] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:47.817734 6189 raft_consensus.cc:3037] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:47.817845 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.814172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:36929 (local address 127.3.222.1:51029)
0928 17:46:47.814347 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.814357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.814448 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.814620 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.814621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.814623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:47.814700 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.814702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.815607 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.815608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.816720 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.816723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.817417 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.817418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.817494 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.817498 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.817584 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.817623 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260928 17:46:47.818127 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c99c6ecf3243ffbe5f0fb036179a17" candidate_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60d6ee16cc7428b9609ac9114addc7c" is_pre_election: true
I20260928 17:46:47.818221 6117 raft_consensus.cc:2454] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0104f7b5ccb4bbab642e7df533bb2b8 in term 0.
I20260928 17:46:47.818619 6189 raft_consensus.cc:504] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.818744 6189 leader_election.cc:288] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [CANDIDATE]: Term 1 election: Requested vote from peers 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141), a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929)
I20260928 17:46:47.818905 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c99c6ecf3243ffbe5f0fb036179a17" candidate_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60d6ee16cc7428b9609ac9114addc7c"
I20260928 17:46:47.818934 5977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c99c6ecf3243ffbe5f0fb036179a17" candidate_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e85c18831e4a648c28dc6d8cbcefe8"
I20260928 17:46:47.818984 6117 raft_consensus.cc:3037] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:47.819000 5977 raft_consensus.cc:3037] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:47.819749 5977 raft_consensus.cc:2454] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0104f7b5ccb4bbab642e7df533bb2b8 in term 1.
I20260928 17:46:47.819752 6117 raft_consensus.cc:2454] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0104f7b5ccb4bbab642e7df533bb2b8 in term 1.
I20260928 17:46:47.819888 5768 leader_election.cc:302] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e85c18831e4a648c28dc6d8cbcefe8, b0104f7b5ccb4bbab642e7df533bb2b8; no voters:
I20260928 17:46:47.819973 6189 raft_consensus.cc:2781] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:47.820113 6189 raft_consensus.cc:686] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 LEADER]: Becoming Leader. State: Replica: b0104f7b5ccb4bbab642e7df533bb2b8, State: Running, Role: LEADER
I20260928 17:46:47.820181 6189 consensus_queue.cc:237] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:47.820845 5679 catalog_manager.cc:5495] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1). New cstate: current_term: 1 leader_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } health_report { overall_health: UNKNOWN } } }
I20260928 17:46:47.848042 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.845821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.3.222.1:46811)
0928 17:46:47.845859 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.845861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.845869 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:46:47.845932 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.845933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.845952 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.846015 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:47.846131 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.846620 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.847364 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.847563 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.847614 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.847752 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.847854 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.847916 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.847951 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.847970 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:46:47.852322 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.850149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:41837 (local address 127.3.222.3:36929)
0928 17:46:47.850200 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.850201 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.850264 (+ 63us) server_negotiation.cc:407] Connection header received
0928 17:46:47.850316 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.850317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.850330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.850399 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:47.850512 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.850967 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.851722 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.851920 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.851972 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.852112 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.852143 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.852181 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.852235 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.852254 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:46:47.852344 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.850070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:36929 (local address 127.3.222.1:41837)
0928 17:46:47.850203 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.850213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.850271 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.850402 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.850403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.850405 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:46:47.850464 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.850465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.851032 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.851033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.851651 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.851653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.851975 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.851976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.852032 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.852044 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.852187 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.852189 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.852224 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.852268 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260928 17:46:47.854198 6117 raft_consensus.cc:1261] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Refusing update from remote peer b0104f7b5ccb4bbab642e7df533bb2b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:47.854262 5977 raft_consensus.cc:1261] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Refusing update from remote peer b0104f7b5ccb4bbab642e7df533bb2b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:47.854645 6236 consensus_queue.cc:1035] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:47.854765 6189 consensus_queue.cc:1035] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:47.861011 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.859063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:49715 (local address 127.3.222.1:46811)
0928 17:46:47.859112 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:47.859114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:47.859122 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:46:47.859141 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:47.859142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:47.859150 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:47.859202 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:47.859328 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.859796 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.860570 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.860746 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.860793 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.860874 (+ 81us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.860896 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.860923 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:47.860953 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 17:46:47.860966 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:46:47.861028 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:47.858961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:46811 (local address 127.3.222.3:49715)
0928 17:46:47.859012 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:47.859022 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:47.859060 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:47.859232 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:47.859233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:47.859235 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:46:47.859274 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.859276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.859863 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.859863 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.860506 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:47.860509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:47.860798 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:47.860799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:47.860831 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:47.860836 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:46:47.860923 (+ 87us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:46:47.860924 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:46:47.860950 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 17:46:47.860980 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 17:46:47.861392 6212 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 17:46:49.379925 5679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:49.380030 5679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:49.380774 5679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298: Sending DeleteTablet for 3 replicas of tablet a5c99c6ecf3243ffbe5f0fb036179a17
I20260928 17:46:49.380992 5817 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c99c6ecf3243ffbe5f0fb036179a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:49 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:55348
I20260928 17:46:49.381024 5957 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c99c6ecf3243ffbe5f0fb036179a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:49 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:41612
I20260928 17:46:49.381068 6097 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c99c6ecf3243ffbe5f0fb036179a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:49 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:41580
I20260928 17:46:49.381233 6267 tablet_replica.cc:330] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: stopping tablet replica
I20260928 17:46:49.381309 6267 raft_consensus.cc:2229] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:49.381352 6267 raft_consensus.cc:2258] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:49.381356 6266 tablet_replica.cc:330] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: stopping tablet replica
I20260928 17:46:49.381402 6266 raft_consensus.cc:2229] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:49.381438 6266 raft_consensus.cc:2258] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:49.381596 6267 ts_tablet_manager.cc:1905] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:49.381672 6266 ts_tablet_manager.cc:1905] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:49.383109 6265 tablet_replica.cc:330] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: stopping tablet replica
I20260928 17:46:49.383180 6265 raft_consensus.cc:2229] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:46:49.383311 6265 raft_consensus.cc:2258] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:49.383507 6267 ts_tablet_manager.cc:1918] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 17:46:49.383544 6265 ts_tablet_manager.cc:1905] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:49.383625 6267 log.cc:1196] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/a5c99c6ecf3243ffbe5f0fb036179a17
I20260928 17:46:49.383872 6267 ts_tablet_manager.cc:1939] T a5c99c6ecf3243ffbe5f0fb036179a17 P a60d6ee16cc7428b9609ac9114addc7c: Deleting consensus metadata
I20260928 17:46:49.384210 5666 catalog_manager.cc:4841] TS a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929): tablet a5c99c6ecf3243ffbe5f0fb036179a17 (table test-table [id=cf27361833db4c519916c8f85a6918a1]) successfully deleted
I20260928 17:46:49.384840 6266 ts_tablet_manager.cc:1918] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 17:46:49.384888 6266 log.cc:1196] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/a5c99c6ecf3243ffbe5f0fb036179a17
I20260928 17:46:49.385064 6266 ts_tablet_manager.cc:1939] T a5c99c6ecf3243ffbe5f0fb036179a17 P 65e85c18831e4a648c28dc6d8cbcefe8: Deleting consensus metadata
I20260928 17:46:49.385064 6265 ts_tablet_manager.cc:1918] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 17:46:49.385128 6265 log.cc:1196] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/a5c99c6ecf3243ffbe5f0fb036179a17
I20260928 17:46:49.385347 5664 catalog_manager.cc:4841] TS 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141): tablet a5c99c6ecf3243ffbe5f0fb036179a17 (table test-table [id=cf27361833db4c519916c8f85a6918a1]) successfully deleted
I20260928 17:46:49.385386 6265 ts_tablet_manager.cc:1939] T a5c99c6ecf3243ffbe5f0fb036179a17 P b0104f7b5ccb4bbab642e7df533bb2b8: Deleting consensus metadata
I20260928 17:46:49.385653 5665 catalog_manager.cc:4841] TS b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811): tablet a5c99c6ecf3243ffbe5f0fb036179a17 (table test-table [id=cf27361833db4c519916c8f85a6918a1]) successfully deleted
Sep 28 17:46:49 dist-test-slave-w3bm krb5kdc[5635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:46:49 dist-test-slave-w3bm krb5kdc[5635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:49.404342 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:49.397699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.3.222.62:35357)
0928 17:46:49.397820 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:49.397822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:49.397831 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:46:49.397887 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:49.397887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:49.397898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:49.397955 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:49.398471 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.398975 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.399712 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.399924 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.399976 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:49.402130 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:49.402132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:49.402132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:49.402139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:49.403378 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:49.403663 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:49.403664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:49.403665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:49.403720 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:49.403875 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:49.403876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:49.403876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:49.403980 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:49.404071 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:49.404171 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 17:46:49.404248 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260928 17:46:49.405541 5679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:46:49.405692 5679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:49.406857 5817 tablet_service.cc:1467] Processing CreateTablet for tablet ef2af3609cc04fdc923e0657debfc800 (DEFAULT_TABLE table=test-table [id=1390d2c55b664e87bc24208321f1b38b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:49.406873 6097 tablet_service.cc:1467] Processing CreateTablet for tablet ef2af3609cc04fdc923e0657debfc800 (DEFAULT_TABLE table=test-table [id=1390d2c55b664e87bc24208321f1b38b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:49.406872 5957 tablet_service.cc:1467] Processing CreateTablet for tablet ef2af3609cc04fdc923e0657debfc800 (DEFAULT_TABLE table=test-table [id=1390d2c55b664e87bc24208321f1b38b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:49.406983 5957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2af3609cc04fdc923e0657debfc800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:49.406983 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2af3609cc04fdc923e0657debfc800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:49.407003 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2af3609cc04fdc923e0657debfc800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:49.408003 6276 tablet_bootstrap.cc:492] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: Bootstrap starting.
I20260928 17:46:49.408289 6277 tablet_bootstrap.cc:492] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: Bootstrap starting.
I20260928 17:46:49.408380 6276 tablet_bootstrap.cc:654] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:49.408419 6278 tablet_bootstrap.cc:492] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: Bootstrap starting.
I20260928 17:46:49.408754 6278 tablet_bootstrap.cc:654] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:49.408946 6276 tablet_bootstrap.cc:492] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: No bootstrap required, opened a new log
I20260928 17:46:49.408999 6276 ts_tablet_manager.cc:1397] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:49.409109 6276 raft_consensus.cc:348] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.409178 6276 raft_consensus.cc:374] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:49.409198 6276 raft_consensus.cc:729] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e85c18831e4a648c28dc6d8cbcefe8, State: Initialized, Role: FOLLOWER
I20260928 17:46:49.409238 6276 consensus_queue.cc:260] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.409255 6277 tablet_bootstrap.cc:654] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:49.409277 6278 tablet_bootstrap.cc:492] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: No bootstrap required, opened a new log
I20260928 17:46:49.409317 6278 ts_tablet_manager.cc:1397] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:49.409406 6276 ts_tablet_manager.cc:1428] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:49.409405 6278 raft_consensus.cc:348] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.409461 6278 raft_consensus.cc:374] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:49.409478 6278 raft_consensus.cc:729] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60d6ee16cc7428b9609ac9114addc7c, State: Initialized, Role: FOLLOWER
I20260928 17:46:49.409557 6278 consensus_queue.cc:260] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.409664 6278 ts_tablet_manager.cc:1428] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:49.409747 6277 tablet_bootstrap.cc:492] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: No bootstrap required, opened a new log
I20260928 17:46:49.409798 6277 ts_tablet_manager.cc:1397] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:49.409929 6277 raft_consensus.cc:348] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.410001 6277 raft_consensus.cc:374] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:49.410036 6277 raft_consensus.cc:729] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0104f7b5ccb4bbab642e7df533bb2b8, State: Initialized, Role: FOLLOWER
I20260928 17:46:49.410075 6277 consensus_queue.cc:260] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.410159 6277 ts_tablet_manager.cc:1428] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:49.488193 6247 raft_consensus.cc:482] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:49.488281 6247 raft_consensus.cc:504] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.488448 6247 leader_election.cc:288] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811), 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141)
I20260928 17:46:49.488646 5837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2af3609cc04fdc923e0657debfc800" candidate_uuid: "a60d6ee16cc7428b9609ac9114addc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" is_pre_election: true
I20260928 17:46:49.488679 5977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2af3609cc04fdc923e0657debfc800" candidate_uuid: "a60d6ee16cc7428b9609ac9114addc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" is_pre_election: true
I20260928 17:46:49.488744 5837 raft_consensus.cc:2454] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60d6ee16cc7428b9609ac9114addc7c in term 0.
I20260928 17:46:49.488754 5977 raft_consensus.cc:2454] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60d6ee16cc7428b9609ac9114addc7c in term 0.
I20260928 17:46:49.488909 6048 leader_election.cc:302] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e85c18831e4a648c28dc6d8cbcefe8, a60d6ee16cc7428b9609ac9114addc7c; no voters:
I20260928 17:46:49.489029 6247 raft_consensus.cc:2781] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:49.489076 6247 raft_consensus.cc:482] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:49.489090 6247 raft_consensus.cc:3037] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:49.489751 6247 raft_consensus.cc:504] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.489864 6247 leader_election.cc:288] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [CANDIDATE]: Term 1 election: Requested vote from peers b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811), 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141)
I20260928 17:46:49.490010 5977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2af3609cc04fdc923e0657debfc800" candidate_uuid: "a60d6ee16cc7428b9609ac9114addc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e85c18831e4a648c28dc6d8cbcefe8"
I20260928 17:46:49.490079 5977 raft_consensus.cc:3037] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:49.490039 5837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2af3609cc04fdc923e0657debfc800" candidate_uuid: "a60d6ee16cc7428b9609ac9114addc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8"
I20260928 17:46:49.490105 5837 raft_consensus.cc:3037] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:49.490706 5977 raft_consensus.cc:2454] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60d6ee16cc7428b9609ac9114addc7c in term 1.
I20260928 17:46:49.490762 5837 raft_consensus.cc:2454] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60d6ee16cc7428b9609ac9114addc7c in term 1.
I20260928 17:46:49.490869 6048 leader_election.cc:302] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e85c18831e4a648c28dc6d8cbcefe8, a60d6ee16cc7428b9609ac9114addc7c; no voters:
I20260928 17:46:49.490953 6247 raft_consensus.cc:2781] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:49.491019 6247 raft_consensus.cc:686] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 1 LEADER]: Becoming Leader. State: Replica: a60d6ee16cc7428b9609ac9114addc7c, State: Running, Role: LEADER
I20260928 17:46:49.491068 6247 consensus_queue.cc:237] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.491667 5679 catalog_manager.cc:5495] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3). New cstate: current_term: 1 leader_uuid: "a60d6ee16cc7428b9609ac9114addc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } health_report { overall_health: HEALTHY } } }
I20260928 17:46:49.503750 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:49.501511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.3.222.3:36929)
0928 17:46:49.501652 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:49.501653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:49.501660 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:46:49.501679 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:49.501679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:49.501688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:49.501741 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:49.501884 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.502366 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.503093 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.503347 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.503401 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:49.503534 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:49.503569 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:49.503613 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:49.503655 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 17:46:49.503675 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260928 17:46:49.508073 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:49.505944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:34885 (local address 127.3.222.3:36929)
0928 17:46:49.505977 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:49.505979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:49.506066 (+ 87us) server_negotiation.cc:407] Connection header received
0928 17:46:49.506099 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:49.506100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:49.506108 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:49.506152 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:49.506294 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.506724 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.507479 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.507671 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.507731 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:49.507881 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:49.507903 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:49.507934 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:49.507989 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 17:46:49.508005 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:46:49.508134 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:49.505800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:36929 (local address 127.3.222.3:34885)
0928 17:46:49.505982 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:49.506007 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:49.506060 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:49.506206 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:49.506206 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:49.506208 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:46:49.506244 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:49.506245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.506793 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.506794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:49.507412 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:49.507414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.507760 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.507761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:49.507818 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:49.507822 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:46:49.507956 (+ 134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:46:49.507957 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:46:49.507987 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 17:46:49.508025 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260928 17:46:49.509711 5977 raft_consensus.cc:1261] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Refusing update from remote peer a60d6ee16cc7428b9609ac9114addc7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:49.509716 5837 raft_consensus.cc:1261] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Refusing update from remote peer a60d6ee16cc7428b9609ac9114addc7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:49.509979 6215 consensus_queue.cc:1035] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:49.510067 6247 consensus_queue.cc:1035] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:49.519692 5680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:49.519780 5680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:49.520358 5680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298: Sending DeleteTablet for 3 replicas of tablet ef2af3609cc04fdc923e0657debfc800
I20260928 17:46:49.520668 6097 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2af3609cc04fdc923e0657debfc800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:49 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:41580
I20260928 17:46:49.520671 5817 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2af3609cc04fdc923e0657debfc800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:49 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:55348
I20260928 17:46:49.520761 6267 tablet_replica.cc:330] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: stopping tablet replica
I20260928 17:46:49.520773 5957 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2af3609cc04fdc923e0657debfc800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:49 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:41612
I20260928 17:46:49.520805 6267 raft_consensus.cc:2229] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:46:49.520853 6266 tablet_replica.cc:330] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: stopping tablet replica
I20260928 17:46:49.520857 6265 tablet_replica.cc:330] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: stopping tablet replica
I20260928 17:46:49.520875 6267 raft_consensus.cc:2258] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:49.520905 6265 raft_consensus.cc:2229] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:49.520908 6266 raft_consensus.cc:2229] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:49.520937 6265 raft_consensus.cc:2258] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:49.520939 6266 raft_consensus.cc:2258] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:49.521082 5680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:46:49.521215 5680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:49.521243 6266 ts_tablet_manager.cc:1905] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:49.521389 6267 ts_tablet_manager.cc:1905] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:49.522411 5957 tablet_service.cc:1467] Processing CreateTablet for tablet 71a1a43686ff49648e5256f14058116d (DEFAULT_TABLE table=test-table [id=ef2cd5a9c0c944ffb94f22a71d4d753b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:49.522506 5957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a1a43686ff49648e5256f14058116d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:49.522862 6265 ts_tablet_manager.cc:1905] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:49.523068 6097 tablet_service.cc:1467] Processing CreateTablet for tablet 71a1a43686ff49648e5256f14058116d (DEFAULT_TABLE table=test-table [id=ef2cd5a9c0c944ffb94f22a71d4d753b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:49.523097 6267 ts_tablet_manager.cc:1918] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:46:49.523144 6267 log.cc:1196] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/ef2af3609cc04fdc923e0657debfc800
I20260928 17:46:49.523149 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a1a43686ff49648e5256f14058116d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:49.523622 5817 tablet_service.cc:1467] Processing CreateTablet for tablet 71a1a43686ff49648e5256f14058116d (DEFAULT_TABLE table=test-table [id=ef2cd5a9c0c944ffb94f22a71d4d753b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:49.523749 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a1a43686ff49648e5256f14058116d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:49.524343 6278 tablet_bootstrap.cc:492] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: Bootstrap starting.
I20260928 17:46:49.524384 6265 ts_tablet_manager.cc:1918] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:46:49.524427 6265 log.cc:1196] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/ef2af3609cc04fdc923e0657debfc800
I20260928 17:46:49.524602 6265 ts_tablet_manager.cc:1939] T ef2af3609cc04fdc923e0657debfc800 P b0104f7b5ccb4bbab642e7df533bb2b8: Deleting consensus metadata
I20260928 17:46:49.524638 6278 tablet_bootstrap.cc:654] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:49.524919 6266 ts_tablet_manager.cc:1918] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:46:49.524951 6277 tablet_bootstrap.cc:492] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8: Bootstrap starting.
I20260928 17:46:49.524964 6266 log.cc:1196] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/ef2af3609cc04fdc923e0657debfc800
I20260928 17:46:49.525135 6278 tablet_bootstrap.cc:492] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: No bootstrap required, opened a new log
I20260928 17:46:49.525153 6266 ts_tablet_manager.cc:1939] T ef2af3609cc04fdc923e0657debfc800 P 65e85c18831e4a648c28dc6d8cbcefe8: Deleting consensus metadata
I20260928 17:46:49.525178 6278 ts_tablet_manager.cc:1397] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:49.525226 6277 tablet_bootstrap.cc:654] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:49.525303 6278 raft_consensus.cc:348] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.525363 6278 raft_consensus.cc:374] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:49.525381 6278 raft_consensus.cc:729] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60d6ee16cc7428b9609ac9114addc7c, State: Initialized, Role: FOLLOWER
I20260928 17:46:49.525415 6278 consensus_queue.cc:260] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.525435 5664 catalog_manager.cc:4841] TS 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2:34141): tablet ef2af3609cc04fdc923e0657debfc800 (table test-table [id=1390d2c55b664e87bc24208321f1b38b]) successfully deleted
I20260928 17:46:49.525480 6267 ts_tablet_manager.cc:1939] T ef2af3609cc04fdc923e0657debfc800 P a60d6ee16cc7428b9609ac9114addc7c: Deleting consensus metadata
I20260928 17:46:49.525494 6278 ts_tablet_manager.cc:1428] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:49.525748 5666 catalog_manager.cc:4841] TS a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929): tablet ef2af3609cc04fdc923e0657debfc800 (table test-table [id=1390d2c55b664e87bc24208321f1b38b]) successfully deleted
I20260928 17:46:49.526145 6276 tablet_bootstrap.cc:492] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8: Bootstrap starting.
I20260928 17:46:49.526187 5665 catalog_manager.cc:4841] TS b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811): tablet ef2af3609cc04fdc923e0657debfc800 (table test-table [id=1390d2c55b664e87bc24208321f1b38b]) successfully deleted
I20260928 17:46:49.526407 6276 tablet_bootstrap.cc:654] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:49.526597 6277 tablet_bootstrap.cc:492] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8: No bootstrap required, opened a new log
I20260928 17:46:49.526636 6277 ts_tablet_manager.cc:1397] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 17:46:49.526734 6277 raft_consensus.cc:348] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.526791 6277 raft_consensus.cc:374] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:49.526819 6277 raft_consensus.cc:729] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0104f7b5ccb4bbab642e7df533bb2b8, State: Initialized, Role: FOLLOWER
I20260928 17:46:49.526855 6277 consensus_queue.cc:260] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.526932 6277 ts_tablet_manager.cc:1428] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:49.526943 6276 tablet_bootstrap.cc:492] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8: No bootstrap required, opened a new log
I20260928 17:46:49.526984 6276 ts_tablet_manager.cc:1397] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 17:46:49.527069 6276 raft_consensus.cc:348] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.527137 6276 raft_consensus.cc:374] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:49.527156 6276 raft_consensus.cc:729] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e85c18831e4a648c28dc6d8cbcefe8, State: Initialized, Role: FOLLOWER
I20260928 17:46:49.527195 6276 consensus_queue.cc:260] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.527297 6276 ts_tablet_manager.cc:1428] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:49.616768 6281 raft_consensus.cc:482] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:49.616888 6281 raft_consensus.cc:504] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.617023 6281 leader_election.cc:288] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811), a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929)
I20260928 17:46:49.617249 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a1a43686ff49648e5256f14058116d" candidate_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60d6ee16cc7428b9609ac9114addc7c" is_pre_election: true
I20260928 17:46:49.617251 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a1a43686ff49648e5256f14058116d" candidate_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" is_pre_election: true
I20260928 17:46:49.617342 6117 raft_consensus.cc:2454] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e85c18831e4a648c28dc6d8cbcefe8 in term 0.
I20260928 17:46:49.617342 5836 raft_consensus.cc:2454] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e85c18831e4a648c28dc6d8cbcefe8 in term 0.
I20260928 17:46:49.617496 5909 leader_election.cc:302] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e85c18831e4a648c28dc6d8cbcefe8, b0104f7b5ccb4bbab642e7df533bb2b8; no voters:
I20260928 17:46:49.617672 6281 raft_consensus.cc:2781] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:49.617728 6281 raft_consensus.cc:482] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:49.617746 6281 raft_consensus.cc:3037] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:49.618443 6281 raft_consensus.cc:504] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.618583 6281 leader_election.cc:288] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [CANDIDATE]: Term 1 election: Requested vote from peers b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811), a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929)
I20260928 17:46:49.618736 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a1a43686ff49648e5256f14058116d" candidate_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60d6ee16cc7428b9609ac9114addc7c"
I20260928 17:46:49.618746 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a1a43686ff49648e5256f14058116d" candidate_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8"
I20260928 17:46:49.618813 6117 raft_consensus.cc:3037] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:49.618824 5836 raft_consensus.cc:3037] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:49.619426 6117 raft_consensus.cc:2454] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e85c18831e4a648c28dc6d8cbcefe8 in term 1.
I20260928 17:46:49.619593 5910 leader_election.cc:302] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e85c18831e4a648c28dc6d8cbcefe8, a60d6ee16cc7428b9609ac9114addc7c; no voters:
I20260928 17:46:49.619635 5836 raft_consensus.cc:2454] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e85c18831e4a648c28dc6d8cbcefe8 in term 1.
I20260928 17:46:49.619680 6281 raft_consensus.cc:2781] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:49.619796 6281 raft_consensus.cc:686] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [term 1 LEADER]: Becoming Leader. State: Replica: 65e85c18831e4a648c28dc6d8cbcefe8, State: Running, Role: LEADER
I20260928 17:46:49.619853 6281 consensus_queue.cc:237] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } }
I20260928 17:46:49.620376 5679 catalog_manager.cc:5495] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65e85c18831e4a648c28dc6d8cbcefe8 (127.3.222.2). New cstate: current_term: 1 leader_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65e85c18831e4a648c28dc6d8cbcefe8" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 34141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 } health_report { overall_health: UNKNOWN } } }
I20260928 17:46:49.646229 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:49.644302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.3.222.2:34141)
0928 17:46:49.644444 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:49.644446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:49.644453 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:46:49.644473 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:49.644473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:49.644482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:49.644534 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:49.644613 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.645091 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.645756 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:49.645931 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:49.645971 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:49.646058 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:49.646080 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:49.646106 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:49.646140 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 17:46:49.646152 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260928 17:46:49.648409 5836 raft_consensus.cc:1261] T 71a1a43686ff49648e5256f14058116d P b0104f7b5ccb4bbab642e7df533bb2b8 [term 1 FOLLOWER]: Refusing update from remote peer 65e85c18831e4a648c28dc6d8cbcefe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:49.648411 6117 raft_consensus.cc:1261] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Refusing update from remote peer 65e85c18831e4a648c28dc6d8cbcefe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:49.648592 6281 consensus_queue.cc:1035] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60d6ee16cc7428b9609ac9114addc7c" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:49.648694 6295 consensus_queue.cc:1035] T 71a1a43686ff49648e5256f14058116d P 65e85c18831e4a648c28dc6d8cbcefe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0104f7b5ccb4bbab642e7df533bb2b8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 17:46:49.653067 6248 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 17:46:51.168463 5678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:51.168560 5678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:51.169287 5678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5faff593dc64d52a7634a9aa96d6298: Sending DeleteTablet for 3 replicas of tablet 71a1a43686ff49648e5256f14058116d
I20260928 17:46:51.169521 6097 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a1a43686ff49648e5256f14058116d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:51 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:41580
I20260928 17:46:51.169524 5957 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a1a43686ff49648e5256f14058116d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:51 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:41612
I20260928 17:46:51.169647 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 5752
W20260928 17:46:51.175196 5665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:46811 (error 108)
I20260928 17:46:51.175349 6323 tablet_replica.cc:330] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: stopping tablet replica
I20260928 17:46:51.175406 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 5892
I20260928 17:46:51.175422 6323 raft_consensus.cc:2229] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:51.175457 6323 raft_consensus.cc:2258] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:51.176127 6323 ts_tablet_manager.cc:1905] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:51.176996 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:51.176806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:46811 (local address 127.0.0.1:38928)
0928 17:46:51.176917 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:51.176960 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
W20260928 17:46:51.177114 5665 catalog_manager.cc:4584] TS b0104f7b5ccb4bbab642e7df533bb2b8 (127.3.222.1:46811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71a1a43686ff49648e5256f14058116d: Network error: Client connection negotiation failed: client connection to 127.3.222.1:46811: connect: Connection refused (error 111)
I20260928 17:46:51.177835 6323 ts_tablet_manager.cc:1918] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 17:46:51.177894 6323 log.cc:1196] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/71a1a43686ff49648e5256f14058116d
I20260928 17:46:51.178082 6323 ts_tablet_manager.cc:1939] T 71a1a43686ff49648e5256f14058116d P a60d6ee16cc7428b9609ac9114addc7c: Deleting consensus metadata
I20260928 17:46:51.178388 5666 catalog_manager.cc:4841] TS a60d6ee16cc7428b9609ac9114addc7c (127.3.222.3:36929): tablet 71a1a43686ff49648e5256f14058116d (table test-table [id=ef2cd5a9c0c944ffb94f22a71d4d753b]) successfully deleted
I20260928 17:46:51.180603 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 6032
I20260928 17:46:51.180655 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:51.180560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:34141 (local address 127.0.0.1:34846)
0928 17:46:51.180604 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:51.180627 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.2:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 17:46:51.185070 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 5648
2026-09-28T17:46:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5472 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:46:51 dist-test-slave-w3bm krb5kdc[6329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:46:51 dist-test-slave-w3bm krb5kdc[6329](info): set up 2 sockets
Sep 28 17:46:51 dist-test-slave-w3bm krb5kdc[6329](info): commencing operation
krb5kdc: starting...
W20260928 17:46:52.134723 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:46:52.245795 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41145
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:52.332479 6342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:52.332665 6342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:52.332686 6342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:52.333739 6342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:52.333772 6342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:52.333784 6342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:52.333794 6342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:52.333813 6342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:52.335186 6342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:52.335400 6342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:52.335566 6342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:52.340029 6347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:52.340059 6348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:52.340082 6350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:52.340220 6342 server_base.cc:1034] running on GCE node
I20260928 17:46:52.340446 6342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:52.340672 6342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:52.341826 6342 hybrid_clock.cc:648] HybridClock initialized: now 1790617612341810 us; error 30 us; skew 500 ppm
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:52.344159 6342 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:46:52.344543 6342 webserver.cc:460] Webserver started at http://127.3.222.62:41327/ using document root <none> and password file <none>
I20260928 17:46:52.344751 6342 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:52.344795 6342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:52.344913 6342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:52.345881 6342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "0ffe8beb399a4c5dac68d12703b1b8dd"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "d8da262eeda4741ed5c79a0d999ccd8d"
server_key_iv: "381895891c6f3e26106e980075d3f387"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.346192 6342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "0ffe8beb399a4c5dac68d12703b1b8dd"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "d8da262eeda4741ed5c79a0d999ccd8d"
server_key_iv: "381895891c6f3e26106e980075d3f387"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.347430 6342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:52.348086 6357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.348253 6342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:52.348347 6342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "0ffe8beb399a4c5dac68d12703b1b8dd"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "d8da262eeda4741ed5c79a0d999ccd8d"
server_key_iv: "381895891c6f3e26106e980075d3f387"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.348405 6342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:52.359495 6342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:52.361248 6342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:52.361375 6342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:52.365144 6342 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:41145
I20260928 17:46:52.365182 6409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:41145 every 8 connection(s)
I20260928 17:46:52.365514 6342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:52.366485 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:52.369697 6410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd: Bootstrap starting.
I20260928 17:46:52.370373 6410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:52.370378 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 6342
I20260928 17:46:52.370450 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:52.370623 3960 external_mini_cluster.cc:1428] Setting key f2f00c04c78e5e34ffedb027b3b6e7a7
I20260928 17:46:52.371482 6410 log.cc:824] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:52.372134 6410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd: No bootstrap required, opened a new log
I20260928 17:46:52.374415 6410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffe8beb399a4c5dac68d12703b1b8dd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41145 } }
I20260928 17:46:52.374576 6410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:52.374616 6410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffe8beb399a4c5dac68d12703b1b8dd, State: Initialized, Role: FOLLOWER
I20260928 17:46:52.374747 6410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffe8beb399a4c5dac68d12703b1b8dd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41145 } }
I20260928 17:46:52.374812 6410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:52.374857 6410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:52.374897 6410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:52.375547 6410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffe8beb399a4c5dac68d12703b1b8dd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41145 } }
I20260928 17:46:52.375650 6410 leader_election.cc:302] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ffe8beb399a4c5dac68d12703b1b8dd; no voters:
I20260928 17:46:52.375799 6410 leader_election.cc:288] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:52.375836 6415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:52.375967 6415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [term 1 LEADER]: Becoming Leader. State: Replica: 0ffe8beb399a4c5dac68d12703b1b8dd, State: Running, Role: LEADER
I20260928 17:46:52.376053 6410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:52.376055 6415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffe8beb399a4c5dac68d12703b1b8dd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41145 } }
I20260928 17:46:52.376441 6415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ffe8beb399a4c5dac68d12703b1b8dd. Latest consensus state: current_term: 1 leader_uuid: "0ffe8beb399a4c5dac68d12703b1b8dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffe8beb399a4c5dac68d12703b1b8dd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41145 } } }
I20260928 17:46:52.376506 6415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:52.376657 6416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ffe8beb399a4c5dac68d12703b1b8dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffe8beb399a4c5dac68d12703b1b8dd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41145 } } }
I20260928 17:46:52.376705 6416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:52.377023 6421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:52.377486 6421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:46:52.378940 6421 catalog_manager.cc:1347] Generated new cluster ID: 5cd5452bce0c43c1b59006c352a8ee6f
I20260928 17:46:52.378990 6421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:52.384961 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:52.371148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.3.222.62:41145)
0928 17:46:52.371575 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:52.371585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:52.371619 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:46:52.372117 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:52.372127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:52.372184 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:52.372307 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:52.373040 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.374425 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.375490 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.376417 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.376872 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:52.381296 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:52.381300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:52.381301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:52.381311 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:52.383035 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:52.383405 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:52.383408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:52.383408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:52.383552 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:52.383737 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:52.383738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:52.383739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:52.384016 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:52.384152 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:52.384624 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 17:46:52.384750 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":79,"threads_started":1}
I20260928 17:46:52.388756 6421 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:52.389511 6421 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:46:52.400938 6421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ffe8beb399a4c5dac68d12703b1b8dd: Generated new TSK 0
I20260928 17:46:52.401125 6421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:46:52.430500 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41145
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:52.508715 6436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:52.508903 6436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:52.508924 6436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:52.510035 6436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:52.510072 6436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:52.510097 6436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:46:52.511482 6436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:52.511675 6436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:52.511873 6436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:46:52.516517 6436 server_base.cc:1034] running on GCE node
W20260928 17:46:52.516618 6444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:52.516736 6442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:52.516491 6441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:52.516984 6436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:52.517187 6436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:52.518316 6436 hybrid_clock.cc:648] HybridClock initialized: now 1790617612518308 us; error 25 us; skew 500 ppm
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:52.520615 6436 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:46:52.521001 6436 webserver.cc:460] Webserver started at http://127.3.222.1:39857/ using document root <none> and password file <none>
I20260928 17:46:52.521196 6436 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:52.521242 6436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:52.521360 6436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:52.522324 6436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "6c0261c4e4b84ca2870be3230c590fd0"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "1c077f35541bcaf1fa6448ae9869ae02"
server_key_iv: "7710df65e96a908718f25d24a5b06977"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.522636 6436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "6c0261c4e4b84ca2870be3230c590fd0"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "1c077f35541bcaf1fa6448ae9869ae02"
server_key_iv: "7710df65e96a908718f25d24a5b06977"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.523837 6436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:46:52.524542 6451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.524699 6436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:46:52.524768 6436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "6c0261c4e4b84ca2870be3230c590fd0"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "1c077f35541bcaf1fa6448ae9869ae02"
server_key_iv: "7710df65e96a908718f25d24a5b06977"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.524827 6436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:52.535382 6436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:52.537037 6436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:52.537153 6436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:52.537330 6436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:52.537596 6436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:52.537626 6436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.537652 6436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:52.537678 6436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.542785 6436 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:38717
I20260928 17:46:52.542845 6564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:38717 every 8 connection(s)
I20260928 17:46:52.543095 6436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:46:52.544410 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 6436
I20260928 17:46:52.544488 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:46:52.544631 3960 external_mini_cluster.cc:1428] Setting key 362d551f7e31e0dbd04e6284b2438428
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:52.552495 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:52.543724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:44021 (local address 127.3.222.62:41145)
0928 17:46:52.543777 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:52.543779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:52.543942 (+ 163us) server_negotiation.cc:407] Connection header received
0928 17:46:52.544420 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:52.544422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:52.544439 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:52.544489 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:52.545496 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.546012 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.547397 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.547663 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.547729 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:52.549855 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:52.549857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:52.549857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:52.549865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:52.551288 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:52.551636 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:52.551637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:52.551638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:52.551693 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:52.551886 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:52.551886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:52.551887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:52.552021 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:52.552092 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:52.552315 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 17:46:52.552399 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:46:52.552552 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:52.543689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41145 (local address 127.3.222.1:44021)
0928 17:46:52.543836 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:52.543872 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:52.544332 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:52.544570 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:52.544573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:52.545213 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:52.545415 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:52.545421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.546094 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.546095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:52.547325 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:52.547327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.547741 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.547742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:52.547861 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:52.548303 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:52.548310 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:52.549768 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:52.551361 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:52.551367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:52.551369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:52.551569 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:52.551753 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:52.551754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:52.551754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:52.551836 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:52.552082 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:52.552086 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:52.552196 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 17:46:52.552275 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 17:46:52.552388 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
I20260928 17:46:52.553066 6565 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41145
I20260928 17:46:52.553148 6565 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:52.553300 6565 heartbeater.cc:511] Master 127.3.222.62:41145 requested a full tablet report, sending...
I20260928 17:46:52.553874 6373 ts_manager.cc:194] Registered new tserver with Master: 6c0261c4e4b84ca2870be3230c590fd0 (127.3.222.1:38717)
I20260928 17:46:52.554466 6373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:44021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:46:52.569924 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41145
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:52.646968 6570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:52.647164 6570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:52.647185 6570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:52.648345 6570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:52.648384 6570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:52.648409 6570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:46:52.649760 6570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:52.649958 6570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:52.650154 6570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:52.654793 6576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:52.654798 6578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:52.654986 6570 server_base.cc:1034] running on GCE node
W20260928 17:46:52.654835 6575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:52.655203 6570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:52.655460 6570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:52.656610 6570 hybrid_clock.cc:648] HybridClock initialized: now 1790617612656602 us; error 29 us; skew 500 ppm
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:52.658918 6570 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:46:52.659353 6570 webserver.cc:460] Webserver started at http://127.3.222.2:33933/ using document root <none> and password file <none>
I20260928 17:46:52.659545 6570 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:52.659590 6570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:52.659705 6570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:52.660712 6570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "a0c636d57d064c208b80695469fee4a8"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "d1aeb4f816ecb41cc6786b77e6998854"
server_key_iv: "51489276ee97fe0a12a95326ba06881c"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.661032 6570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "a0c636d57d064c208b80695469fee4a8"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "d1aeb4f816ecb41cc6786b77e6998854"
server_key_iv: "51489276ee97fe0a12a95326ba06881c"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.662189 6570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:52.662909 6585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.663074 6570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:52.663154 6570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "a0c636d57d064c208b80695469fee4a8"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "d1aeb4f816ecb41cc6786b77e6998854"
server_key_iv: "51489276ee97fe0a12a95326ba06881c"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.663234 6570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:52.683071 6570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:52.685184 6570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:52.685316 6570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:52.685526 6570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:52.685806 6570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:52.685838 6570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.685876 6570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:52.685899 6570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.691234 6570 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:34813
I20260928 17:46:52.691274 6698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:34813 every 8 connection(s)
I20260928 17:46:52.691571 6570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:46:52.693938 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 6570
I20260928 17:46:52.694023 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:46:52.694185 3960 external_mini_cluster.cc:1428] Setting key fb849ed23cc69e36ec52415dccb3a27e
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:52.700413 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:52.692178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:37217 (local address 127.3.222.62:41145)
0928 17:46:52.692257 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:52.692259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:52.692415 (+ 156us) server_negotiation.cc:407] Connection header received
0928 17:46:52.692853 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:52.692854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:52.692863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:52.692910 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:52.693753 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.694316 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.695391 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.695592 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.695645 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:52.697769 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:52.697771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:52.697771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:52.697778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:52.699140 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:52.699513 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:52.699514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:52.699514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:52.699576 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:52.699771 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:52.699771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:52.699772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:52.699964 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:52.700028 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:52.700247 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 17:46:52.700325 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 17:46:52.700470 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:52.692153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41145 (local address 127.3.222.2:37217)
0928 17:46:52.692298 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:52.692332 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:52.692782 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:52.692963 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:52.692967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:52.693495 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:52.693683 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:52.693688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.694393 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.694395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:52.695308 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:52.695311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.695653 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.695654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:52.695800 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:52.696251 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:52.696260 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:52.697685 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:52.699237 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:52.699244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:52.699246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:52.699446 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:52.699634 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:52.699635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:52.699635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:52.699720 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:52.700027 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:52.700030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:52.700134 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 17:46:52.700206 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 17:46:52.700313 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260928 17:46:52.700919 6699 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41145
I20260928 17:46:52.700997 6699 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:52.701200 6699 heartbeater.cc:511] Master 127.3.222.62:41145 requested a full tablet report, sending...
I20260928 17:46:52.701548 6373 ts_manager.cc:194] Registered new tserver with Master: a0c636d57d064c208b80695469fee4a8 (127.3.222.2:34813)
I20260928 17:46:52.701884 6373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:37217
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:46:52.718734 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41145
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:52.795693 6704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:52.795888 6704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:52.795918 6704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:52.797043 6704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:52.797086 6704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:52.797119 6704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:46:52.798486 6704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:52.798704 6704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:52.798906 6704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:52.803454 6710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:52.803473 6712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:52.803588 6704 server_base.cc:1034] running on GCE node
W20260928 17:46:52.803462 6709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:52.803824 6704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:52.804039 6704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:52.805179 6704 hybrid_clock.cc:648] HybridClock initialized: now 1790617612805163 us; error 28 us; skew 500 ppm
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:52.807463 6704 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:46:52.807806 6704 webserver.cc:460] Webserver started at http://127.3.222.3:38001/ using document root <none> and password file <none>
I20260928 17:46:52.808028 6704 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:52.808068 6704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:52.808159 6704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:52.809078 6704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "c1bc62378d2440b593abea64f99cb2e5"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "91da20bf3b3971a5ec01a9636518a9ac"
server_key_iv: "41fafcbf84a3991c3453fe4bb4b51508"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.809346 6704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "c1bc62378d2440b593abea64f99cb2e5"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "91da20bf3b3971a5ec01a9636518a9ac"
server_key_iv: "41fafcbf84a3991c3453fe4bb4b51508"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.810474 6704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:46:52.811066 6719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.811228 6704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.811299 6704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "c1bc62378d2440b593abea64f99cb2e5"
format_stamp: "Formatted at 2026-09-28 17:46:52 on dist-test-slave-w3bm"
server_key: "91da20bf3b3971a5ec01a9636518a9ac"
server_key_iv: "41fafcbf84a3991c3453fe4bb4b51508"
server_key_version: "encryptionkey@0"
I20260928 17:46:52.811349 6704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:52.839910 6704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:52.841850 6704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:52.841972 6704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:52.842162 6704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:52.842455 6704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:52.842491 6704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.842512 6704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:52.842527 6704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:52.847738 6704 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:35471
I20260928 17:46:52.847787 6832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:35471 every 8 connection(s)
I20260928 17:46:52.848129 6704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:46:52.852807 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 6704
I20260928 17:46:52.852893 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:46:52.853068 3960 external_mini_cluster.cc:1428] Setting key bbf00a9511135b8fc62b83494f328386
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:52.856622 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:52.848798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:33913 (local address 127.3.222.62:41145)
0928 17:46:52.848857 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:52.848859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:52.849048 (+ 189us) server_negotiation.cc:407] Connection header received
0928 17:46:52.849547 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:52.849548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:52.849559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:52.849608 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:52.850525 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.851049 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.852029 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.852282 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.852338 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:52.854391 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:52.854393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:52.854393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:52.854399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:52.855635 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:52.855969 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:52.855969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:52.855970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:52.856016 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:52.856185 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:52.856186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:52.856186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:52.856275 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:52.856317 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:52.856507 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 17:46:52.856560 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:46:52.856793 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:52.848768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41145 (local address 127.3.222.3:33913)
0928 17:46:52.848933 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:52.848968 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:52.849451 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:52.849675 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:52.849678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:52.850238 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:52.850437 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:52.850442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.851120 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.851121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:52.851946 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:52.851948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:52.852349 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:52.852350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:52.852475 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:52.852913 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:52.852923 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:52.854306 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:52.855707 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:52.855712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:52.855714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:52.855914 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:52.856060 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:52.856061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:52.856061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:52.856145 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:52.856314 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:52.856317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:52.856422 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 17:46:52.856497 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:46:52.856608 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260928 17:46:52.857218 6833 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41145
I20260928 17:46:52.857300 6833 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:52.857467 6833 heartbeater.cc:511] Master 127.3.222.62:41145 requested a full tablet report, sending...
I20260928 17:46:52.857792 6373 ts_manager.cc:194] Registered new tserver with Master: c1bc62378d2440b593abea64f99cb2e5 (127.3.222.3:35471)
I20260928 17:46:52.858166 6373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:33913
I20260928 17:46:52.864812 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 17:46:52.874490 6413 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:46:52.873735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.3.222.62:41145)
0928 17:46:52.873778 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:52.873780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:52.873821 (+ 41us) server_negotiation.cc:407] Connection header received
0928 17:46:52.873882 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:52.873883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:52.873892 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:52.873965 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:52.874437 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40832: BlockingRecv error: recv got EOF from 127.0.0.1:40832 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:46:52.875433 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 6436
I20260928 17:46:52.879335 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 6570
I20260928 17:46:52.883076 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 6704
I20260928 17:46:52.886493 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 6342
2026-09-28T17:46:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1700 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6845](info): set up 2 sockets
Sep 28 17:46:52 dist-test-slave-w3bm krb5kdc[6845](info): commencing operation
krb5kdc: starting...
W20260928 17:46:53.839283 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:46:53 dist-test-slave-w3bm krb5kdc[6845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:46:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:46:53.957849 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:43387
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:43387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:54.033859 6858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:54.034056 6858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:54.034085 6858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:54.035235 6858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:54.035274 6858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:54.035291 6858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:54.035308 6858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:54.035323 6858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:54.036778 6858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:43387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:43387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:54.037004 6858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:54.037196 6858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:54.041687 6866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:54.041728 6864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:54.041700 6863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:54.041882 6858 server_base.cc:1034] running on GCE node
I20260928 17:46:54.042166 6858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:54.042433 6858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:54.043596 6858 hybrid_clock.cc:648] HybridClock initialized: now 1790617614043577 us; error 33 us; skew 500 ppm
Sep 28 17:46:54 dist-test-slave-w3bm krb5kdc[6845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:54.045835 6858 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:46:54.046237 6858 webserver.cc:460] Webserver started at http://127.3.222.62:39185/ using document root <none> and password file <none>
I20260928 17:46:54.046403 6858 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:54.046438 6858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:54.046522 6858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:54.047549 6858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5"
format_stamp: "Formatted at 2026-09-28 17:46:54 on dist-test-slave-w3bm"
server_key: "892158de008e7765aacedd135bc63ce9"
server_key_iv: "d5049cd7128b2c45037c9937ab75d55f"
server_key_version: "encryptionkey@0"
I20260928 17:46:54.047819 6858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5"
format_stamp: "Formatted at 2026-09-28 17:46:54 on dist-test-slave-w3bm"
server_key: "892158de008e7765aacedd135bc63ce9"
server_key_iv: "d5049cd7128b2c45037c9937ab75d55f"
server_key_version: "encryptionkey@0"
I20260928 17:46:54.048934 6858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:54.049527 6873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:54.049688 6858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:54.049772 6858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5"
format_stamp: "Formatted at 2026-09-28 17:46:54 on dist-test-slave-w3bm"
server_key: "892158de008e7765aacedd135bc63ce9"
server_key_iv: "d5049cd7128b2c45037c9937ab75d55f"
server_key_version: "encryptionkey@0"
I20260928 17:46:54.049837 6858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:54.065922 6858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:54.067725 6858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:54.067845 6858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:54.071264 6858 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:43387
I20260928 17:46:54.071305 6925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:43387 every 8 connection(s)
I20260928 17:46:54.071616 6858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:54.072423 6926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:54.074635 6926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5: Bootstrap starting.
I20260928 17:46:54.075151 6926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:54.075418 6926 log.cc:824] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:54.076031 6926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5: No bootstrap required, opened a new log
I20260928 17:46:54.077497 6926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43387 } }
I20260928 17:46:54.077636 6926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:54.077667 6926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64c7f1db0871472faf1b1aa1ed1cc4d5, State: Initialized, Role: FOLLOWER
I20260928 17:46:54.077791 6926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43387 } }
I20260928 17:46:54.077863 6926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:54.077893 6926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:54.077937 6926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:54.078668 6926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43387 } }
I20260928 17:46:54.078756 6926 leader_election.cc:302] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64c7f1db0871472faf1b1aa1ed1cc4d5; no voters:
I20260928 17:46:54.078907 6926 leader_election.cc:288] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:54.078958 6929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:54.079098 6929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [term 1 LEADER]: Becoming Leader. State: Replica: 64c7f1db0871472faf1b1aa1ed1cc4d5, State: Running, Role: LEADER
I20260928 17:46:54.079128 6926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:54.079183 6929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43387 } }
I20260928 17:46:54.079509 6930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43387 } } }
I20260928 17:46:54.079530 6931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64c7f1db0871472faf1b1aa1ed1cc4d5. Latest consensus state: current_term: 1 leader_uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c7f1db0871472faf1b1aa1ed1cc4d5" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43387 } } }
I20260928 17:46:54.079599 6930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:54.079604 6931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:54.079793 6938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:54.080245 6938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:46:54.081580 6938 catalog_manager.cc:1347] Generated new cluster ID: a51a8e775adb4d9c8febd92680314482
I20260928 17:46:54.081626 6938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:46:54.081629 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 6858
I20260928 17:46:54.081717 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:54.081882 3960 external_mini_cluster.cc:1428] Setting key a30b72f42aa45d4f80e4f73971ec16c3
Sep 28 17:46:54 dist-test-slave-w3bm krb5kdc[6845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:54.097146 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:54.082294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.3.222.62:43387)
0928 17:46:54.086974 (+ 4680us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:54.086984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:54.087021 (+ 37us) server_negotiation.cc:407] Connection header received
0928 17:46:54.087153 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:54.087160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:54.087246 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:54.087399 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:54.088073 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:54.089504 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:54.090485 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:54.091118 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:54.091242 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:54.093807 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:54.093811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:54.093812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:54.093824 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:54.095576 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:54.095936 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:54.095939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:54.095939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:54.096046 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:54.096214 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:54.096215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:54.096215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:54.096490 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:54.096628 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:54.096718 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 17:46:54.096928 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4620,"thread_start_us":60,"threads_started":1}
I20260928 17:46:54.097975 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:46:54.108871 6938 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:54.109864 6938 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:46:54.113013 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:54.109061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.3.222.62:43387)
0928 17:46:54.109097 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:54.109099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:54.109213 (+ 114us) server_negotiation.cc:407] Connection header received
0928 17:46:54.109303 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:54.109304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:54.109314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:54.109342 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:54.109802 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:54.110500 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:54.111505 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:54.111844 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:54.111920 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:54.112559 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:54.112560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:54.112560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:46:54.112764 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:54.112783 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:46:54.112790 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:54.112842 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:54.112895 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 17:46:54.112935 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:46:54.119716 6938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64c7f1db0871472faf1b1aa1ed1cc4d5: Generated new TSK 0
I20260928 17:46:54.119884 6938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:46:54.125648 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 6858
2026-09-28T17:46:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1237 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:46:54 dist-test-slave-w3bm krb5kdc[6959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:46:54 dist-test-slave-w3bm krb5kdc[6959](info): set up 2 sockets
Sep 28 17:46:54 dist-test-slave-w3bm krb5kdc[6959](info): commencing operation
krb5kdc: starting...
W20260928 17:46:55.070739 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:46:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:46:55.182614 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:33025
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:33025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:55.259333 6972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:55.259512 6972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:55.259532 6972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:55.260936 6972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:55.260975 6972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:55.260994 6972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:55.261011 6972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:55.261030 6972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:55.262463 6972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:33025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:55.262676 6972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:55.262879 6972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:55.267262 6977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:55.267256 6978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:55.267256 6980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:55.267488 6972 server_base.cc:1034] running on GCE node
I20260928 17:46:55.267699 6972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:55.267935 6972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:55.269063 6972 hybrid_clock.cc:648] HybridClock initialized: now 1790617615269048 us; error 30 us; skew 500 ppm
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:55.271420 6972 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:46:55.271821 6972 webserver.cc:460] Webserver started at http://127.3.222.62:40731/ using document root <none> and password file <none>
I20260928 17:46:55.272015 6972 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:55.272056 6972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:55.272173 6972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:55.273155 6972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "64d380d6d207d35375d7e1010fffa006"
server_key_iv: "2559337be3df85164b2744a260a5255a"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.273465 6972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "64d380d6d207d35375d7e1010fffa006"
server_key_iv: "2559337be3df85164b2744a260a5255a"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.274600 6972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:46:55.275182 6987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.275343 6972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.275421 6972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "64d380d6d207d35375d7e1010fffa006"
server_key_iv: "2559337be3df85164b2744a260a5255a"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.275485 6972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:55.293589 6972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:55.295527 6972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:55.295650 6972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:55.299316 6972 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:33025
I20260928 17:46:55.299348 7039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:33025 every 8 connection(s)
I20260928 17:46:55.299647 6972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:55.300522 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:55.302632 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f: Bootstrap starting.
I20260928 17:46:55.303141 7040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:55.303404 7040 log.cc:824] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:55.304008 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f: No bootstrap required, opened a new log
I20260928 17:46:55.305485 7040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33025 } }
I20260928 17:46:55.305614 7040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:55.305646 7040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d4e27cd91b4ba6bddd3a5b48ea136f, State: Initialized, Role: FOLLOWER
I20260928 17:46:55.305781 7040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33025 } }
I20260928 17:46:55.305860 7040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:55.305893 7040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:55.305939 7040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:55.306643 7040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33025 } }
I20260928 17:46:55.306740 7040 leader_election.cc:302] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37d4e27cd91b4ba6bddd3a5b48ea136f; no voters:
I20260928 17:46:55.306820 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 6972
I20260928 17:46:55.306885 7040 leader_election.cc:288] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:55.306913 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:55.306905 7043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:55.307073 3960 external_mini_cluster.cc:1428] Setting key 4ef9aafcf82df9795ffdcb2b25d58a2c
I20260928 17:46:55.307071 7043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [term 1 LEADER]: Becoming Leader. State: Replica: 37d4e27cd91b4ba6bddd3a5b48ea136f, State: Running, Role: LEADER
I20260928 17:46:55.307147 7043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33025 } }
I20260928 17:46:55.307173 7040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:55.307449 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d4e27cd91b4ba6bddd3a5b48ea136f. Latest consensus state: current_term: 1 leader_uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33025 } } }
I20260928 17:46:55.307505 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:55.307629 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d4e27cd91b4ba6bddd3a5b48ea136f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33025 } } }
I20260928 17:46:55.307746 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f [sys.catalog]: This master's current role is: LEADER
W20260928 17:46:55.308765 7060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:46:55.308816 7060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:46:55.308866 7050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:55.309067 7050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:46:55.310509 7050 catalog_manager.cc:1347] Generated new cluster ID: 08a826f9dea747ccbdc709921ac913e2
I20260928 17:46:55.310559 7050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:55.315413 7050 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:55.316048 7050 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:46:55.316805 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:55.307497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.3.222.62:33025)
0928 17:46:55.307888 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:55.307897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:55.307927 (+ 30us) server_negotiation.cc:407] Connection header received
0928 17:46:55.308036 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:55.308043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:55.308096 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:55.308245 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:55.308935 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.310009 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.310979 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.311412 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.311502 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:55.313488 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:55.313492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:55.313492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:55.313503 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:55.315451 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.315752 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.315755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.315756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.315847 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.316005 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.316006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.316006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.316243 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:55.316401 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:55.316474 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 17:46:55.316605 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":60,"threads_started":1}
I20260928 17:46:55.328449 7050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d4e27cd91b4ba6bddd3a5b48ea136f: Generated new TSK 0
I20260928 17:46:55.328637 7050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:46:55.358446 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:33025
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:55.436743 7066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:55.436934 7066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:55.436955 7066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:55.438072 7066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:55.438108 7066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:55.438133 7066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:46:55.439515 7066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:55.439715 7066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:55.439913 7066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:55.444633 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:55.444693 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:55.444725 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:55.444824 7066 server_base.cc:1034] running on GCE node
I20260928 17:46:55.444995 7066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:55.445217 7066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:55.446381 7066 hybrid_clock.cc:648] HybridClock initialized: now 1790617615446361 us; error 23 us; skew 500 ppm
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:55.448693 7066 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:46:55.449075 7066 webserver.cc:460] Webserver started at http://127.3.222.1:42311/ using document root <none> and password file <none>
I20260928 17:46:55.449258 7066 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:55.449302 7066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:55.449450 7066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:55.450462 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "e3cbfded5a9646c99e578e23ca7e0f3d"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "a48446e470e1803eab362c436e6511d2"
server_key_iv: "bd6a9e35747f8e4f19906f7134e502c8"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.450781 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "e3cbfded5a9646c99e578e23ca7e0f3d"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "a48446e470e1803eab362c436e6511d2"
server_key_iv: "bd6a9e35747f8e4f19906f7134e502c8"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.452003 7066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:55.452669 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.452806 7066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:55.452888 7066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "e3cbfded5a9646c99e578e23ca7e0f3d"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "a48446e470e1803eab362c436e6511d2"
server_key_iv: "bd6a9e35747f8e4f19906f7134e502c8"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.452942 7066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:55.476289 7066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:55.478099 7066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:55.478219 7066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:55.478413 7066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:55.478652 7066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:55.478679 7066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.478698 7066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:55.478714 7066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.484087 7066 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:39869
I20260928 17:46:55.484124 7194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:39869 every 8 connection(s)
I20260928 17:46:55.484485 7066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:55.492836 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:55.485116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:40069 (local address 127.3.222.62:33025)
0928 17:46:55.485158 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:55.485160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:55.485342 (+ 182us) server_negotiation.cc:407] Connection header received
0928 17:46:55.485832 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:55.485833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:55.485845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:55.485909 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:55.486800 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.487357 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.488335 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.488542 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.488597 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:55.490567 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:55.490568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:55.490569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:55.490576 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:55.491669 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.492006 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.492007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.492008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.492060 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.492257 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.492258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.492259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.492365 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:55.492423 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:55.492658 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 17:46:55.492746 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:46:55.492974 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:55.485103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:33025 (local address 127.3.222.1:40069)
0928 17:46:55.485235 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:55.485268 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:55.485742 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:55.485980 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:55.485984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:55.486526 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:55.486709 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:55.486714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.487441 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.487443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:55.488260 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:55.488262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.488606 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.488608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:55.488732 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:55.489179 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:55.489187 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:55.490484 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:55.491736 (+ 1252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:55.491742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:55.491744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:55.491944 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:55.492110 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:55.492111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:55.492111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:55.492203 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:55.492418 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:55.492423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:55.492548 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 17:46:55.492634 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 17:46:55.492774 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
I20260928 17:46:55.492962 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 7066
I20260928 17:46:55.493047 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:46:55.493224 3960 external_mini_cluster.cc:1428] Setting key 8eae6cce5acbaa14811c0669444f3bf8
I20260928 17:46:55.493973 7195 heartbeater.cc:347] Connected to a master server at 127.3.222.62:33025
I20260928 17:46:55.494065 7195 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:55.494222 7195 heartbeater.cc:511] Master 127.3.222.62:33025 requested a full tablet report, sending...
I20260928 17:46:55.494709 7003 ts_manager.cc:194] Registered new tserver with Master: e3cbfded5a9646c99e578e23ca7e0f3d (127.3.222.1:39869)
I20260928 17:46:55.495518 7003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:40069
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:46:55.515829 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:33025
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:55.593115 7200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:55.593322 7200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:55.593344 7200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:55.594436 7200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:55.594466 7200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:55.594489 7200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:46:55.595857 7200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:55.596027 7200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:55.596190 7200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:55.600724 7206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:55.600822 7200 server_base.cc:1034] running on GCE node
W20260928 17:46:55.600780 7205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:55.600714 7208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:55.601135 7200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:55.601343 7200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:55.602484 7200 hybrid_clock.cc:648] HybridClock initialized: now 1790617615602470 us; error 29 us; skew 500 ppm
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:55.604822 7200 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:46:55.605208 7200 webserver.cc:460] Webserver started at http://127.3.222.2:40249/ using document root <none> and password file <none>
I20260928 17:46:55.605410 7200 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:55.605456 7200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:55.605571 7200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:55.606525 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "25ba825c6ffa47e7aa5eaef3e044cad7"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "1ac27fecdee74b83df3955c99fcd3833"
server_key_iv: "176c11c91c77de6be7660503b540265b"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.606827 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "25ba825c6ffa47e7aa5eaef3e044cad7"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "1ac27fecdee74b83df3955c99fcd3833"
server_key_iv: "176c11c91c77de6be7660503b540265b"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.608119 7200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:46:55.608733 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.608891 7200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:46:55.608969 7200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "25ba825c6ffa47e7aa5eaef3e044cad7"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "1ac27fecdee74b83df3955c99fcd3833"
server_key_iv: "176c11c91c77de6be7660503b540265b"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.609030 7200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:55.626547 7200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:55.628216 7200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:55.628337 7200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:55.628523 7200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:55.628772 7200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:55.628803 7200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.628829 7200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:55.628852 7200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.634037 7200 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:45001
I20260928 17:46:55.634089 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:45001 every 8 connection(s)
I20260928 17:46:55.634346 7200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:55.639842 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 7200
I20260928 17:46:55.639925 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:46:55.640089 3960 external_mini_cluster.cc:1428] Setting key 30e855c6f4cd61a9f5137fe3b5e71219
I20260928 17:46:55.642714 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:55.634966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:46133 (local address 127.3.222.62:33025)
0928 17:46:55.635012 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:55.635014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:55.635182 (+ 168us) server_negotiation.cc:407] Connection header received
0928 17:46:55.635699 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:55.635700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:55.635710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:55.635747 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:55.636574 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.637057 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.637947 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.638116 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.638159 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:55.640109 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:55.640110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:55.640111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:55.640119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:55.641506 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.641858 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.641860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.641860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.641916 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.642165 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.642166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.642166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.642273 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:55.642324 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:55.642598 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 17:46:55.642652 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:46:55.642940 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:55.634938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:33025 (local address 127.3.222.2:46133)
0928 17:46:55.635076 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:55.635108 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:55.635609 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:55.635800 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:55.635803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:55.636322 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:55.636509 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:55.636513 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.637114 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.637115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:55.637891 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:55.637892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.638184 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.638184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:55.638277 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:55.638725 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:55.638733 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:55.640032 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:55.641579 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:55.641583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:55.641585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:55.641774 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:55.641986 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:55.641987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:55.641987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:55.642112 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:55.642326 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:55.642329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:55.642482 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 17:46:55.642585 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 17:46:55.642733 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260928 17:46:55.643419 7329 heartbeater.cc:347] Connected to a master server at 127.3.222.62:33025
I20260928 17:46:55.643507 7329 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:55.643666 7329 heartbeater.cc:511] Master 127.3.222.62:33025 requested a full tablet report, sending...
I20260928 17:46:55.643957 7003 ts_manager.cc:194] Registered new tserver with Master: 25ba825c6ffa47e7aa5eaef3e044cad7 (127.3.222.2:45001)
I20260928 17:46:55.644297 7003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:46133
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:46:55.662930 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:33025
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:55.738901 7334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:55.739111 7334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:55.739143 7334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:55.740360 7334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:55.740402 7334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:55.740437 7334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:46:55.741807 7334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:55.742017 7334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:55.742213 7334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:55.746762 7339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:55.746737 7340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:55.746762 7334 server_base.cc:1034] running on GCE node
W20260928 17:46:55.746726 7342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:55.747099 7334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:55.747337 7334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:55.748469 7334 hybrid_clock.cc:648] HybridClock initialized: now 1790617615748454 us; error 35 us; skew 500 ppm
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:55.750706 7334 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:46:55.751091 7334 webserver.cc:460] Webserver started at http://127.3.222.3:37671/ using document root <none> and password file <none>
I20260928 17:46:55.751315 7334 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:55.751361 7334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:55.751480 7334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:55.752436 7334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "1b1dd48186df418a9ff33c4a3448aeef"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "51c1d48258a4bd22ea597fa1e8f41f0d"
server_key_iv: "88e6eed955447c549fe1d3cca059e22e"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.752739 7334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "1b1dd48186df418a9ff33c4a3448aeef"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "51c1d48258a4bd22ea597fa1e8f41f0d"
server_key_iv: "88e6eed955447c549fe1d3cca059e22e"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.753896 7334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:55.754532 7349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.754670 7334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:55.754755 7334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "1b1dd48186df418a9ff33c4a3448aeef"
format_stamp: "Formatted at 2026-09-28 17:46:55 on dist-test-slave-w3bm"
server_key: "51c1d48258a4bd22ea597fa1e8f41f0d"
server_key_iv: "88e6eed955447c549fe1d3cca059e22e"
server_key_version: "encryptionkey@0"
I20260928 17:46:55.754812 7334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:55.764671 7334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:55.766535 7334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:55.766659 7334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:55.766846 7334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:55.767099 7334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:55.767129 7334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.767156 7334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:55.767179 7334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:55.772414 7334 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:38471
I20260928 17:46:55.772467 7462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:38471 every 8 connection(s)
I20260928 17:46:55.772728 7334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:46:55.776522 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 7334
I20260928 17:46:55.776615 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:46:55.776788 3960 external_mini_cluster.cc:1428] Setting key 7bebfea8728e9708c073558bc2de3527
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:55.781597 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:55.773330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:34003 (local address 127.3.222.62:33025)
0928 17:46:55.773427 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:55.773429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:55.773522 (+ 93us) server_negotiation.cc:407] Connection header received
0928 17:46:55.773982 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:55.773983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:55.773992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:55.774026 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:55.774879 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.775521 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.776518 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.776797 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.776863 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:55.779037 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:55.779038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:55.779038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:55.779044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:55.780401 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.780795 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.780796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.780796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.780851 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.781062 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.781062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.781063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.781167 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:55.781224 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:55.781474 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 17:46:55.781534 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 17:46:55.781778 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:55.773295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:33025 (local address 127.3.222.3:34003)
0928 17:46:55.773420 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:55.773452 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:55.773898 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:55.774086 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:55.774089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:55.774600 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:55.774784 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:55.774788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.775624 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.775625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:55.776426 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:55.776428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.776893 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.776894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:55.777033 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:55.777590 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:55.777598 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:55.778960 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:55.780485 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:55.780491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:55.780492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:55.780696 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:55.780910 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:55.780910 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:55.780911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:55.781005 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:55.781243 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:55.781248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:55.781361 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 17:46:55.781448 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 17:46:55.781570 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
I20260928 17:46:55.782179 7463 heartbeater.cc:347] Connected to a master server at 127.3.222.62:33025
I20260928 17:46:55.782302 7463 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:55.782433 7463 heartbeater.cc:511] Master 127.3.222.62:33025 requested a full tablet report, sending...
I20260928 17:46:55.782755 7003 ts_manager.cc:194] Registered new tserver with Master: 1b1dd48186df418a9ff33c4a3448aeef (127.3.222.3:38471)
I20260928 17:46:55.783067 7003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:34003
I20260928 17:46:55.788556 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[6959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617615, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:55.809401 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:55.802774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.3.222.62:33025)
0928 17:46:55.802847 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:55.802849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:55.802907 (+ 58us) server_negotiation.cc:407] Connection header received
0928 17:46:55.802953 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:55.802953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:55.802962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:55.803015 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:55.803606 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.804111 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.804848 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:55.805022 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:55.805064 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:55.807145 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:55.807147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:55.807147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:55.807154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:55.808402 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.808708 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.808710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.808710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.808759 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:55.808924 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:55.808925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:55.808925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:55.809056 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:55.809140 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:55.809238 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 17:46:55.809309 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 17:46:55.809618 7003 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39292
I20260928 17:46:55.810012 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 7066
I20260928 17:46:55.814402 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 7200
I20260928 17:46:55.818091 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 7334
I20260928 17:46:55.821590 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 6972
2026-09-28T17:46:55Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1697 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[7475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[7475](info): set up 2 sockets
Sep 28 17:46:55 dist-test-slave-w3bm krb5kdc[7475](info): commencing operation
krb5kdc: starting...
W20260928 17:46:56.784967 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 17:46:56.944562 3960 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.159s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:46:57.036036 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:34231
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:34231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:57.113101 7488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:57.113324 7488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:57.113353 7488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:57.114635 7488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:57.114689 7488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:57.114710 7488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:57.114729 7488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:57.114754 7488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:57.116413 7488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:34231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:34231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:57.116636 7488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:57.116833 7488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:57.121417 7493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:57.121578 7488 server_base.cc:1034] running on GCE node
W20260928 17:46:57.121420 7494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:57.121417 7496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:57.121912 7488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:57.122162 7488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:57.123257 7488 hybrid_clock.cc:648] HybridClock initialized: now 1790617617123246 us; error 26 us; skew 500 ppm
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:57.132927 7488 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:46:57.133378 7488 webserver.cc:460] Webserver started at http://127.3.222.62:38825/ using document root <none> and password file <none>
I20260928 17:46:57.133589 7488 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:57.133636 7488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:57.133760 7488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:57.134794 7488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "397b426ff96547a09ba4f634162e27aa"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "ed6f8f8e5fd8db8eeb1d259c1f0ff9c1"
server_key_iv: "b554b63b75302bef2350f4f9635ce225"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.135109 7488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "397b426ff96547a09ba4f634162e27aa"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "ed6f8f8e5fd8db8eeb1d259c1f0ff9c1"
server_key_iv: "b554b63b75302bef2350f4f9635ce225"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.136325 7488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:46:57.136993 7503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.137148 7488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:46:57.137228 7488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "397b426ff96547a09ba4f634162e27aa"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "ed6f8f8e5fd8db8eeb1d259c1f0ff9c1"
server_key_iv: "b554b63b75302bef2350f4f9635ce225"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.137288 7488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:57.164623 7488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:57.166321 7488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:57.166453 7488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:57.170408 7488 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:34231
I20260928 17:46:57.170433 7555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:34231 every 8 connection(s)
I20260928 17:46:57.170917 7488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:57.171561 7556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:57.173677 7556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa: Bootstrap starting.
I20260928 17:46:57.174176 7556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:57.174392 7556 log.cc:824] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:57.174973 7556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa: No bootstrap required, opened a new log
I20260928 17:46:57.176455 7556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397b426ff96547a09ba4f634162e27aa" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34231 } }
I20260928 17:46:57.176592 7556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:57.176614 7556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397b426ff96547a09ba4f634162e27aa, State: Initialized, Role: FOLLOWER
I20260928 17:46:57.176715 7556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397b426ff96547a09ba4f634162e27aa" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34231 } }
I20260928 17:46:57.176764 7556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:57.176783 7556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:57.176815 7556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:57.177593 7556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397b426ff96547a09ba4f634162e27aa" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34231 } }
I20260928 17:46:57.177675 7556 leader_election.cc:302] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 397b426ff96547a09ba4f634162e27aa; no voters:
I20260928 17:46:57.177788 7556 leader_election.cc:288] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:57.177836 7559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:57.177989 7559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [term 1 LEADER]: Becoming Leader. State: Replica: 397b426ff96547a09ba4f634162e27aa, State: Running, Role: LEADER
I20260928 17:46:57.178004 7556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:57.178155 7559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397b426ff96547a09ba4f634162e27aa" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34231 } }
I20260928 17:46:57.178498 7560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "397b426ff96547a09ba4f634162e27aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397b426ff96547a09ba4f634162e27aa" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34231 } } }
I20260928 17:46:57.178586 7560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:57.178774 7567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:57.178797 7561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 397b426ff96547a09ba4f634162e27aa. Latest consensus state: current_term: 1 leader_uuid: "397b426ff96547a09ba4f634162e27aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397b426ff96547a09ba4f634162e27aa" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34231 } } }
I20260928 17:46:57.179111 7561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:57.179327 7567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:46:57.180039 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 7488
I20260928 17:46:57.180130 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:57.180305 3960 external_mini_cluster.cc:1428] Setting key c745a5a475f2f1a4c1370fb63525d3eb
I20260928 17:46:57.183585 7567 catalog_manager.cc:1347] Generated new cluster ID: fadae306fa8244a38ef1ca3a6795021b
I20260928 17:46:57.183645 7567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:57.191905 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:57.180714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.3.222.62:34231)
0928 17:46:57.180872 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:57.180880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:57.183694 (+ 2814us) server_negotiation.cc:407] Connection header received
0928 17:46:57.183884 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:57.183890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:57.183932 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:57.184041 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:57.184605 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.185786 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.186501 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.186935 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.187024 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:57.189145 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:57.189149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:57.189149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:57.189159 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:57.190545 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.190840 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.190844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.190844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.190921 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.191084 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.191085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.191085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.191341 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:57.191477 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:57.191581 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 17:46:57.191709 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260928 17:46:57.199672 7567 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:57.200130 7567 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:46:57.207404 7567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 397b426ff96547a09ba4f634162e27aa: Generated new TSK 0
I20260928 17:46:57.207582 7567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:46:57.237633 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:34231
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:57.316519 7582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:57.316710 7582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:57.316740 7582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:57.317858 7582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:57.317894 7582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:57.317927 7582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:46:57.319408 7582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:57.319629 7582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:57.319819 7582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:57.324457 7587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:57.324512 7590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:57.324555 7582 server_base.cc:1034] running on GCE node
W20260928 17:46:57.324481 7588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:57.324810 7582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:57.324997 7582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:57.326128 7582 hybrid_clock.cc:648] HybridClock initialized: now 1790617617326116 us; error 26 us; skew 500 ppm
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:57.328356 7582 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:46:57.328732 7582 webserver.cc:460] Webserver started at http://127.3.222.1:35337/ using document root <none> and password file <none>
I20260928 17:46:57.328900 7582 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:57.328938 7582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:57.329018 7582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:57.329937 7582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "1417d448f5ce45e29c4202df2bc07f5b"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "2dfee32f81e4d1fda3bb4326c190004a"
server_key_iv: "2b4a824abc211e02df9bacb3870d1f20"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.330206 7582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "1417d448f5ce45e29c4202df2bc07f5b"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "2dfee32f81e4d1fda3bb4326c190004a"
server_key_iv: "2b4a824abc211e02df9bacb3870d1f20"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.331321 7582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:57.331969 7597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.332124 7582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:57.332195 7582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "1417d448f5ce45e29c4202df2bc07f5b"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "2dfee32f81e4d1fda3bb4326c190004a"
server_key_iv: "2b4a824abc211e02df9bacb3870d1f20"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.332244 7582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:57.347920 7582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:57.349663 7582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:57.349763 7582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:57.349929 7582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:57.350180 7582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:57.350212 7582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.350231 7582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:57.350255 7582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.355425 7582 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:45237
I20260928 17:46:57.355486 7710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:45237 every 8 connection(s)
I20260928 17:46:57.355751 7582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:57.362228 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 7582
I20260928 17:46:57.362314 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:46:57.362476 3960 external_mini_cluster.cc:1428] Setting key 07d4c905abcefbd78991690cebba2a60
I20260928 17:46:57.364329 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:57.356384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:52495 (local address 127.3.222.62:34231)
0928 17:46:57.356447 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:57.356450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:57.356623 (+ 173us) server_negotiation.cc:407] Connection header received
0928 17:46:57.357581 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:57.357582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:57.357595 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:57.357647 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:57.358198 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.358703 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.359674 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.359908 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.359963 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:57.361824 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:57.361825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:57.361826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:57.361833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:57.363129 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.363501 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.363502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.363503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.363564 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.363760 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.363761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.363762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.363864 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:57.363916 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:57.364162 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 17:46:57.364242 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:46:57.364475 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:57.356332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:34231 (local address 127.3.222.1:52495)
0928 17:46:57.356509 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:57.356544 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:57.357441 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:57.357708 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:57.357711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:57.357938 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:57.358131 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:57.358135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.358764 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.358765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:57.359600 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:57.359602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.359997 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.359999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:57.360111 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:57.360487 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:57.360494 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:57.361731 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:57.363196 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:57.363203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:57.363205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:57.363436 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:57.363614 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:57.363615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:57.363616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:57.363707 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:57.363913 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:57.363919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:57.364049 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 17:46:57.364132 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 17:46:57.364273 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260928 17:46:57.364876 7711 heartbeater.cc:347] Connected to a master server at 127.3.222.62:34231
I20260928 17:46:57.364969 7711 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:57.365126 7711 heartbeater.cc:511] Master 127.3.222.62:34231 requested a full tablet report, sending...
I20260928 17:46:57.365588 7520 ts_manager.cc:194] Registered new tserver with Master: 1417d448f5ce45e29c4202df2bc07f5b (127.3.222.1:45237)
I20260928 17:46:57.366407 7520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:52495
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:46:57.383896 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:34231
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:57.460412 7716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:57.460605 7716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:57.460635 7716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:57.461725 7716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:57.461762 7716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:57.461795 7716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:46:57.463203 7716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:57.463428 7716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:57.463615 7716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:57.468238 7722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:57.468274 7716 server_base.cc:1034] running on GCE node
W20260928 17:46:57.468255 7724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:57.468245 7721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:57.468611 7716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:57.468834 7716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:57.469987 7716 hybrid_clock.cc:648] HybridClock initialized: now 1790617617469977 us; error 26 us; skew 500 ppm
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:57.472189 7716 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:46:57.472604 7716 webserver.cc:460] Webserver started at http://127.3.222.2:35201/ using document root <none> and password file <none>
I20260928 17:46:57.472787 7716 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:57.472831 7716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:57.472944 7716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:57.473903 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "ccbd1952df444b3e9cae9c788ca5ddc9"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "9b1b8831768cacd645de20902139da44"
server_key_iv: "5e2e3c561ca4e7479592d4d495930327"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.474206 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "ccbd1952df444b3e9cae9c788ca5ddc9"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "9b1b8831768cacd645de20902139da44"
server_key_iv: "5e2e3c561ca4e7479592d4d495930327"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.475366 7716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:57.475970 7731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.476109 7716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.476181 7716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "ccbd1952df444b3e9cae9c788ca5ddc9"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "9b1b8831768cacd645de20902139da44"
server_key_iv: "5e2e3c561ca4e7479592d4d495930327"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.476243 7716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:57.500476 7716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:57.502244 7716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:57.502368 7716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:57.502581 7716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:57.502859 7716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:57.502892 7716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.502919 7716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:57.502943 7716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.508265 7716 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:40493
I20260928 17:46:57.508322 7844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:40493 every 8 connection(s)
I20260928 17:46:57.508620 7716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:57.516790 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:57.509279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:42801 (local address 127.3.222.62:34231)
0928 17:46:57.509325 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:57.509327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:57.509501 (+ 174us) server_negotiation.cc:407] Connection header received
0928 17:46:57.510365 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:57.510366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:57.510375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:57.510426 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:57.511017 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.511559 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.512487 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.512687 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.512737 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:57.514554 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:57.514555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:57.514555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:57.514561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:57.515775 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.516099 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.516099 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.516099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.516154 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.516318 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.516319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.516319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.516410 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:57.516455 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:57.516669 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 17:46:57.516726 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:46:57.516950 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:57.509247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:34231 (local address 127.3.222.2:42801)
0928 17:46:57.509386 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:57.509429 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:57.510287 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:57.510480 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:57.510484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:57.510733 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:57.510918 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:57.510924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.511627 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.511628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:57.512412 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:57.512414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.512743 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.512743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:57.512833 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:57.513250 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:57.513258 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:57.514477 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:57.515837 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:57.515842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:57.515844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:57.516047 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:57.516194 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:57.516195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:57.516195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:57.516277 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:57.516451 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:57.516453 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:57.516575 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 17:46:57.516650 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:46:57.516757 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260928 17:46:57.517400 7845 heartbeater.cc:347] Connected to a master server at 127.3.222.62:34231
I20260928 17:46:57.517484 7845 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:57.517614 7845 heartbeater.cc:511] Master 127.3.222.62:34231 requested a full tablet report, sending...
I20260928 17:46:57.517963 7520 ts_manager.cc:194] Registered new tserver with Master: ccbd1952df444b3e9cae9c788ca5ddc9 (127.3.222.2:40493)
I20260928 17:46:57.518028 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 7716
I20260928 17:46:57.518131 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:46:57.518280 3960 external_mini_cluster.cc:1428] Setting key b131a21b5ca686fc6ff40aba0b13f06e
I20260928 17:46:57.518409 7520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:42801
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:46:57.540232 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:34231
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:57.618224 7850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:57.618404 7850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:57.618426 7850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:57.619551 7850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:57.619588 7850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:57.619612 7850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:46:57.621008 7850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:57.621179 7850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:57.621340 7850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:57.625753 7858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:57.625768 7856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:57.625811 7855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:57.625941 7850 server_base.cc:1034] running on GCE node
I20260928 17:46:57.626155 7850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:57.626356 7850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:57.627494 7850 hybrid_clock.cc:648] HybridClock initialized: now 1790617617627478 us; error 35 us; skew 500 ppm
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:57.629725 7850 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:46:57.630120 7850 webserver.cc:460] Webserver started at http://127.3.222.3:40635/ using document root <none> and password file <none>
I20260928 17:46:57.630317 7850 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:57.630358 7850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:57.630481 7850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:57.631477 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "da4bae3290dd4db9bdae52a3d7c458e5"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "fffd2ee8b694eee4e2b4224797a5c423"
server_key_iv: "360bf510d1e55a2a121ea50a3959525e"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.631780 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "da4bae3290dd4db9bdae52a3d7c458e5"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "fffd2ee8b694eee4e2b4224797a5c423"
server_key_iv: "360bf510d1e55a2a121ea50a3959525e"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.632903 7850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:57.633554 7865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.633683 7850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:57.633764 7850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "da4bae3290dd4db9bdae52a3d7c458e5"
format_stamp: "Formatted at 2026-09-28 17:46:57 on dist-test-slave-w3bm"
server_key: "fffd2ee8b694eee4e2b4224797a5c423"
server_key_iv: "360bf510d1e55a2a121ea50a3959525e"
server_key_version: "encryptionkey@0"
I20260928 17:46:57.633826 7850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:57.643791 7850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:57.645310 7850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:57.645426 7850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:57.645607 7850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:57.645864 7850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:57.645892 7850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.645911 7850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:57.645934 7850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:57.651142 7850 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:39799
I20260928 17:46:57.651182 7978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:39799 every 8 connection(s)
I20260928 17:46:57.651463 7850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:46:57.654491 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 7850
I20260928 17:46:57.654595 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:46:57.654762 3960 external_mini_cluster.cc:1428] Setting key d5d704c29cbec4cec89e086dbd8fee09
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:57.660055 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:57.652139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:41251 (local address 127.3.222.62:34231)
0928 17:46:57.652183 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:57.652185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:57.652380 (+ 195us) server_negotiation.cc:407] Connection header received
0928 17:46:57.653308 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:57.653310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:57.653323 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:57.653370 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:57.653963 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.654462 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.655729 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.655915 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.655968 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:57.657862 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:57.657863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:57.657863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:57.657870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:57.658864 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.659224 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.659226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.659226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.659279 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.659484 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.659485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.659485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.659588 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:57.659646 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:57.659891 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 17:46:57.659968 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:46:57.660215 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:57.652117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:34231 (local address 127.3.222.3:41251)
0928 17:46:57.652276 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:57.652307 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:57.653210 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:57.653436 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:57.653441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:57.653698 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:57.653877 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:57.653882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.654541 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.654542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:57.655660 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:57.655663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.655979 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.655980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:57.656106 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:57.656512 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:57.656520 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:57.657777 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:57.658936 (+ 1159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:57.658942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:57.658944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:57.659146 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:57.659332 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:57.659332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:57.659332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:57.659432 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:57.659646 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:57.659652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:57.659778 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 17:46:57.659862 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:46:57.660002 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260928 17:46:57.660626 7979 heartbeater.cc:347] Connected to a master server at 127.3.222.62:34231
I20260928 17:46:57.660702 7979 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:57.660835 7979 heartbeater.cc:511] Master 127.3.222.62:34231 requested a full tablet report, sending...
I20260928 17:46:57.661149 7520 ts_manager.cc:194] Registered new tserver with Master: da4bae3290dd4db9bdae52a3d7c458e5 (127.3.222.3:39799)
I20260928 17:46:57.661511 7520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:41251
I20260928 17:46:57.666297 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:46:57.686712 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:57.679897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.3.222.1:45237)
0928 17:46:57.680035 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:57.680039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:57.680054 (+ 15us) server_negotiation.cc:407] Connection header received
0928 17:46:57.680077 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:57.680078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:57.680091 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:57.680151 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:57.680663 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.681375 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.682037 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.682329 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.682378 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:57.684308 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:57.684316 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:57.684316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:57.684323 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:57.685656 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.685990 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.685993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.685994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.686050 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.686194 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.686194 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.686194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.686325 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:57.686422 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:57.686559 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 17:46:57.686627 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:57.693845 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:57.687843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.3.222.62:34231)
0928 17:46:57.687883 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:57.687885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:57.687893 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:46:57.687946 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:57.687947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:57.687954 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:57.687991 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:57.688385 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.688859 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.689480 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:57.689657 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:57.689698 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:57.691697 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:57.691698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:57.691699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:57.691706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:57.692885 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.693177 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.693178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.693178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.693230 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:57.693379 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:57.693380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:57.693380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:57.693511 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:57.693597 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:57.693696 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 17:46:57.693759 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:46:57.694042 7520 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53248
I20260928 17:46:57.694325 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 7582
I20260928 17:46:57.698212 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 7716
I20260928 17:46:57.701836 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 7850
I20260928 17:46:57.705477 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 7488
2026-09-28T17:46:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1883 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7991](info): set up 2 sockets
Sep 28 17:46:57 dist-test-slave-w3bm krb5kdc[7991](info): commencing operation
krb5kdc: starting...
W20260928 17:46:58.654904 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:46:58 dist-test-slave-w3bm krb5kdc[7991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:46:58.770349 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35251
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:58.848141 8004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:58.848351 8004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:58.848380 8004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:58.849591 8004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:58.849632 8004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:58.849659 8004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:58.849676 8004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:58.849690 8004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:58.851133 8004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:35251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:58.851383 8004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:58.851572 8004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:58.856101 8009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:58.856112 8010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:58.856172 8012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:58.856153 8004 server_base.cc:1034] running on GCE node
I20260928 17:46:58.856492 8004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:58.856750 8004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:58.857923 8004 hybrid_clock.cc:648] HybridClock initialized: now 1790617618857904 us; error 23 us; skew 500 ppm
Sep 28 17:46:58 dist-test-slave-w3bm krb5kdc[7991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:58.860224 8004 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:46:58.860541 8004 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:58.860590 8004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:58.860711 8004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:58.861732 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "cd4a0fe769e946c2a8f295f929d2253a"
format_stamp: "Formatted at 2026-09-28 17:46:58 on dist-test-slave-w3bm"
server_key: "e7612a50a65052495425d55f5606f2d3"
server_key_iv: "bb9167abac4fa205c7cd8cb379c385dd"
server_key_version: "encryptionkey@0"
I20260928 17:46:58.862047 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "cd4a0fe769e946c2a8f295f929d2253a"
format_stamp: "Formatted at 2026-09-28 17:46:58 on dist-test-slave-w3bm"
server_key: "e7612a50a65052495425d55f5606f2d3"
server_key_iv: "bb9167abac4fa205c7cd8cb379c385dd"
server_key_version: "encryptionkey@0"
I20260928 17:46:58.863333 8004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:58.864054 8018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:58.864230 8004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:58.864323 8004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "cd4a0fe769e946c2a8f295f929d2253a"
format_stamp: "Formatted at 2026-09-28 17:46:58 on dist-test-slave-w3bm"
server_key: "e7612a50a65052495425d55f5606f2d3"
server_key_iv: "bb9167abac4fa205c7cd8cb379c385dd"
server_key_version: "encryptionkey@0"
I20260928 17:46:58.864395 8004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:58.878194 8004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:58.880235 8004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:58.880354 8004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:58.884079 8004 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:35251
I20260928 17:46:58.884105 8070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:35251 every 8 connection(s)
I20260928 17:46:58.884366 8004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:58.884559 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 8004
I20260928 17:46:58.884670 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:58.884841 3960 external_mini_cluster.cc:1428] Setting key cd4b007a8c7a78637e0fff757c2cd8f9
I20260928 17:46:58.885371 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:58.887866 8071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a: Bootstrap starting.
I20260928 17:46:58.888441 8071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:58.888702 8071 log.cc:824] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:58.889293 8071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a: No bootstrap required, opened a new log
Sep 28 17:46:58 dist-test-slave-w3bm krb5kdc[7991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:58.891396 8071 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4a0fe769e946c2a8f295f929d2253a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35251 } }
I20260928 17:46:58.891528 8071 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:58.891561 8071 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd4a0fe769e946c2a8f295f929d2253a, State: Initialized, Role: FOLLOWER
I20260928 17:46:58.891686 8071 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4a0fe769e946c2a8f295f929d2253a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35251 } }
I20260928 17:46:58.891757 8071 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:58.891788 8071 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:58.891834 8071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:58.892303 8071 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4a0fe769e946c2a8f295f929d2253a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35251 } }
I20260928 17:46:58.892395 8071 leader_election.cc:302] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd4a0fe769e946c2a8f295f929d2253a; no voters:
I20260928 17:46:58.892539 8071 leader_election.cc:288] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:58.892580 8076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:58.892715 8076 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [term 1 LEADER]: Becoming Leader. State: Replica: cd4a0fe769e946c2a8f295f929d2253a, State: Running, Role: LEADER
I20260928 17:46:58.892793 8071 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:58.892863 8076 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4a0fe769e946c2a8f295f929d2253a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35251 } }
I20260928 17:46:58.893215 8076 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd4a0fe769e946c2a8f295f929d2253a. Latest consensus state: current_term: 1 leader_uuid: "cd4a0fe769e946c2a8f295f929d2253a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4a0fe769e946c2a8f295f929d2253a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35251 } } }
I20260928 17:46:58.893357 8076 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:58.893971 8085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:58.894443 8078 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd4a0fe769e946c2a8f295f929d2253a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4a0fe769e946c2a8f295f929d2253a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35251 } } }
I20260928 17:46:58.894558 8085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:46:58.894584 8078 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:58.894791 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:58.885275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.3.222.62:35251)
0928 17:46:58.885454 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:58.885465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:58.885497 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:46:58.885946 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:58.885952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:58.885997 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:58.886117 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:58.886709 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:58.887629 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:58.888410 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:58.888886 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:58.888980 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:58.891319 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:58.891324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:58.891325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:58.891336 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:58.892984 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:58.893364 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:58.893367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:58.893368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:58.893463 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:58.893713 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:58.893714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:58.893714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:58.894024 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:58.894157 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:58.894370 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 17:46:58.894494 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260928 17:46:58.896288 8085 catalog_manager.cc:1347] Generated new cluster ID: 3d10f728be2a452f9dbfc83843e8dee9
I20260928 17:46:58.896343 8085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:46:58.910041 8085 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:58.910750 8085 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:46:58.920518 8085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a: Generated new TSK 0
I20260928 17:46:58.920703 8085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:46:58.952538 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35251
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:59.030637 8097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:59.030844 8097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:59.030876 8097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:59.032030 8097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:59.032069 8097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:59.032104 8097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:46:59.033537 8097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:59.033764 8097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:59.033957 8097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:59.038852 8103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.038852 8105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.039121 8102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:59.039150 8097 server_base.cc:1034] running on GCE node
I20260928 17:46:59.039361 8097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:59.039547 8097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:59.040684 8097 hybrid_clock.cc:648] HybridClock initialized: now 1790617619040669 us; error 28 us; skew 500 ppm
Sep 28 17:46:59 dist-test-slave-w3bm krb5kdc[7991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:59.043079 8097 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:46:59.043428 8097 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:59.043479 8097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:59.043602 8097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:59.044608 8097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "494e7a138f814af2832cff742b38ef13"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "59609b59be815c5f4633a9dba330aed3"
server_key_iv: "afd7a3a0e8256683e7484fc77bb4617d"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.044921 8097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "494e7a138f814af2832cff742b38ef13"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "59609b59be815c5f4633a9dba330aed3"
server_key_iv: "afd7a3a0e8256683e7484fc77bb4617d"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.046046 8097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:46:59.046727 8111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.046871 8097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:59.046963 8097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "494e7a138f814af2832cff742b38ef13"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "59609b59be815c5f4633a9dba330aed3"
server_key_iv: "afd7a3a0e8256683e7484fc77bb4617d"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.047019 8097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:59.056133 8097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:59.057832 8097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:59.057932 8097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:59.058099 8097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:59.058341 8097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:59.058370 8097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.058389 8097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:59.058406 8097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.063675 8097 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:41241
I20260928 17:46:59.063736 8224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:41241 every 8 connection(s)
I20260928 17:46:59.063956 8097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:46:59.067389 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 8097
I20260928 17:46:59.067485 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:46:59.067655 3960 external_mini_cluster.cc:1428] Setting key 734ab17394ab76756c1983f1891a84f9
Sep 28 17:46:59 dist-test-slave-w3bm krb5kdc[7991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:59.074239 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.064638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:43815 (local address 127.3.222.62:35251)
0928 17:46:59.064707 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.064710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.064841 (+ 131us) server_negotiation.cc:407] Connection header received
0928 17:46:59.065313 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.065315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.065325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.065371 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:59.066259 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.066754 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.068001 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.068236 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.068292 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.071556 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:59.071557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:59.071558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:59.071565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:59.072862 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:59.073335 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:59.073336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:59.073336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:59.073390 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:59.073642 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:59.073643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:59.073643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:59.073742 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:59.073786 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.074098 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.074168 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 17:46:59.074474 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.064569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35251 (local address 127.3.222.1:43815)
0928 17:46:59.064724 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.064765 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.065232 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.065433 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.065436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.065678 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:59.066145 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.066150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.066840 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.066840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.067918 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.067921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.069379 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.069380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.069485 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.069958 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:59.069966 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:59.071468 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:59.072948 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:59.072955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:59.072958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:59.073256 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:59.073451 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:59.073452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:59.073452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:59.073586 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:59.073799 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:59.073805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:59.073969 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.074090 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.074253 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260928 17:46:59.074975 8225 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35251
I20260928 17:46:59.075057 8225 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:59.075196 8225 heartbeater.cc:511] Master 127.3.222.62:35251 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
I20260928 17:46:59.075740 8035 ts_manager.cc:194] Registered new tserver with Master: 494e7a138f814af2832cff742b38ef13 (127.3.222.1:41241)
I20260928 17:46:59.076324 8035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:43815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:46:59.094429 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35251
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:59.171451 8230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:59.171653 8230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:59.171684 8230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:59.172818 8230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:59.172857 8230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:59.172892 8230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:46:59.174340 8230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:59.174559 8230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:59.174758 8230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:59.179483 8235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.179502 8238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.179527 8236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:59.179589 8230 server_base.cc:1034] running on GCE node
I20260928 17:46:59.179900 8230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:59.180130 8230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:59.181277 8230 hybrid_clock.cc:648] HybridClock initialized: now 1790617619181253 us; error 30 us; skew 500 ppm
Sep 28 17:46:59 dist-test-slave-w3bm krb5kdc[7991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:59.183710 8230 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:46:59.184036 8230 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:59.184085 8230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:59.184231 8230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:59.185220 8230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "1b9df138d9dd4cd5ad7d7aac822314ce"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "c8da3c0d585ff3f7089463a83b7e48e4"
server_key_iv: "87b79f656ae58a463039224f3663c6e2"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.185539 8230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "1b9df138d9dd4cd5ad7d7aac822314ce"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "c8da3c0d585ff3f7089463a83b7e48e4"
server_key_iv: "87b79f656ae58a463039224f3663c6e2"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.186669 8230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:46:59.187325 8244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.187491 8230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.187569 8230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "1b9df138d9dd4cd5ad7d7aac822314ce"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "c8da3c0d585ff3f7089463a83b7e48e4"
server_key_iv: "87b79f656ae58a463039224f3663c6e2"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.187637 8230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:59.220343 8230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:59.222259 8230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:59.222399 8230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:59.222627 8230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:59.222929 8230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:59.222962 8230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.222991 8230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:59.223008 8230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.228382 8230 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:40117
I20260928 17:46:59.228432 8357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:40117 every 8 connection(s)
I20260928 17:46:59.228708 8230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:46:59.229250 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 8230
I20260928 17:46:59.229367 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:46:59.229560 3960 external_mini_cluster.cc:1428] Setting key e2f016277275d9dd22be4982115462ce
Sep 28 17:46:59 dist-test-slave-w3bm krb5kdc[7991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
I20260928 17:46:59.238668 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.229431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:34189 (local address 127.3.222.62:35251)
0928 17:46:59.229483 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.229485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.229693 (+ 208us) server_negotiation.cc:407] Connection header received
0928 17:46:59.230247 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.230248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.230257 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.230301 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:59.231251 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.231833 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.233348 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.233529 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.233581 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.236074 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:59.236075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:59.236075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:59.236083 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:59.237328 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:59.237811 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:59.237813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:59.237813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:59.237868 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:59.238086 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:59.238086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:59.238087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:59.238184 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:59.238228 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.238506 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.238582 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:46:59.238854 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.229360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35251 (local address 127.3.222.2:34189)
0928 17:46:59.229522 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.229577 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.230161 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.230388 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.230391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.230647 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:59.231135 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.231140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.231952 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.231953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.233256 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.233260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.233619 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.233620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.233765 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.234240 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:59.234250 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:59.235958 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:59.237426 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:59.237434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:59.237437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:59.237734 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:59.237935 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:59.237936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:59.237936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:59.238038 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:59.238263 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:59.238269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:59.238392 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.238489 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.238643 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260928 17:46:59.239341 8358 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35251
I20260928 17:46:59.239437 8358 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:59.239631 8358 heartbeater.cc:511] Master 127.3.222.62:35251 requested a full tablet report, sending...
I20260928 17:46:59.240012 8035 ts_manager.cc:194] Registered new tserver with Master: 1b9df138d9dd4cd5ad7d7aac822314ce (127.3.222.2:40117)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
I20260928 17:46:59.240459 8035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:34189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:46:59.254362 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35251
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:46:59.333048 8363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:59.333244 8363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:59.333266 8363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:59.334362 8363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:59.334398 8363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:59.334422 8363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:46:59.335891 8363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:59.336082 8363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:59.336246 8363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:59.340829 8371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.340838 8368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.340837 8369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:59.341053 8363 server_base.cc:1034] running on GCE node
I20260928 17:46:59.341229 8363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:59.341470 8363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:59.342638 8363 hybrid_clock.cc:648] HybridClock initialized: now 1790617619342616 us; error 28 us; skew 500 ppm
Sep 28 17:46:59 dist-test-slave-w3bm krb5kdc[7991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:46:59.344991 8363 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:46:59.345360 8363 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:59.345412 8363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:59.345533 8363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:59.346527 8363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "d4c3ba013a6eff175e4e5307aaeaf67e"
server_key_iv: "40bf2586595fe8fa4ad32bb544b0cf4c"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.346844 8363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "d4c3ba013a6eff175e4e5307aaeaf67e"
server_key_iv: "40bf2586595fe8fa4ad32bb544b0cf4c"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.348088 8363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:46:59.348803 8377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.348963 8363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.349041 8363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "d4c3ba013a6eff175e4e5307aaeaf67e"
server_key_iv: "40bf2586595fe8fa4ad32bb544b0cf4c"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.349100 8363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:59.361009 8363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:59.362799 8363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:59.362946 8363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:59.363154 8363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:59.363451 8363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:59.363483 8363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.363511 8363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:59.363528 8363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.369050 8363 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:46167
I20260928 17:46:59.369081 8490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:46167 every 8 connection(s)
I20260928 17:46:59.369314 8363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:46:59 dist-test-slave-w3bm krb5kdc[7991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:46:59.377789 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.370031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:34279 (local address 127.3.222.62:35251)
0928 17:46:59.370123 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.370126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.370270 (+ 144us) server_negotiation.cc:407] Connection header received
0928 17:46:59.370789 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.370790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.370803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.370854 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:59.371779 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.372273 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.373232 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.373417 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.373470 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.375422 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:59.375423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:59.375424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:59.375431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:59.376590 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:59.376943 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:59.376944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:59.376944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:59.376993 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:59.377206 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:59.377207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:59.377208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:59.377311 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:59.377365 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.377625 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.377702 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 17:46:59.377942 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.369993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35251 (local address 127.3.222.3:34279)
0928 17:46:59.370142 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.370188 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.370692 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.370928 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.370931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.371182 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:59.371689 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.371694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.372355 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.372356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.373150 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.373153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.373487 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.373488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.373594 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.374037 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:46:59.374045 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:59.375341 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:59.376670 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:59.376675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:59.376677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:59.376877 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:59.377047 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:46:59.377047 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:46:59.377048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:46:59.377148 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:46:59.377368 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:59.377374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:59.377510 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.377594 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.377742 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260928 17:46:59.378387 8491 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35251
I20260928 17:46:59.378463 8491 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:59.378604 8491 heartbeater.cc:511] Master 127.3.222.62:35251 requested a full tablet report, sending...
I20260928 17:46:59.378734 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 8363
I20260928 17:46:59.378837 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:46:59.379006 3960 external_mini_cluster.cc:1428] Setting key fee9902b1044d53d7464792d80c0dc54
I20260928 17:46:59.379035 8035 ts_manager.cc:194] Registered new tserver with Master: af8aeb6f301946bbbe5b7b08f23bd2d2 (127.3.222.3:46167)
I20260928 17:46:59.379488 8035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:34279
I20260928 17:46:59.379866 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:46:59.386658 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.381063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.3.222.62:35251)
0928 17:46:59.381116 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.381119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.381198 (+ 79us) server_negotiation.cc:407] Connection header received
0928 17:46:59.381261 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.381262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.381272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.381313 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:59.381851 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.382315 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.383035 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.383273 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.383326 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.384376 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:59.384378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:59.384378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:46:59.384384 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:59.385586 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:59.385931 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:59.385932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:59.385932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:59.385987 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:46:59.386151 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:46:59.386151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:46:59.386152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:46:59.386286 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:59.386373 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.386503 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.386572 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:46:59.388087 8035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:46:59.389722 8035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:46:59.395704 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.391758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:41241 (local address 127.0.0.1:47378)
0928 17:46:59.392234 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.392242 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.392288 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.392453 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.392454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.392456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:59.392497 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.392499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.393479 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.393480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.394614 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.394618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.395400 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.395401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.395477 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.395491 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.395563 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.395614 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":57,"thread_start_us":51,"threads_started":1}
I20260928 17:46:59.395726 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.391997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.3.222.1:41241)
0928 17:46:59.392191 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.392196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.392287 (+ 91us) server_negotiation.cc:407] Connection header received
0928 17:46:59.392356 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.392358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.392372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.392445 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:59.392558 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.393392 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.394719 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.395202 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.395300 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.395368 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.395596 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.395643 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260928 17:46:59.396303 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.391757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:46167 (local address 127.0.0.1:36366)
0928 17:46:59.392128 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.392137 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.392194 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.392642 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.392643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.392645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:59.392694 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.392696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.394016 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.394017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.395170 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.395173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.396074 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.396075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.396134 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.396138 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.396185 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.396225 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"spinlock_wait_cycles":768,"thread_start_us":68,"threads_started":1}
I20260928 17:46:59.396410 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.391757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40117 (local address 127.0.0.1:53184)
0928 17:46:59.391906 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.391943 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.392018 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.392205 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.392208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.392210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:59.392277 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.392279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.393794 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.393795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.395364 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.395366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.396186 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.396187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.396238 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.396240 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.396281 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.396312 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":48,"thread_start_us":49,"threads_started":1}
I20260928 17:46:59.396466 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.391863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.3.222.2:40117)
0928 17:46:59.392044 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.392050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.392060 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:46:59.392100 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.392102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.392118 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.392199 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:59.392329 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.393689 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.395446 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.396037 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.396103 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.396158 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.396349 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.396390 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260928 17:46:59.396576 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.391969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36366 (local address 127.3.222.3:46167)
0928 17:46:59.392402 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.392406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.392414 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:46:59.392446 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.392448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.392462 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.392536 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:59.392754 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.393902 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.395282 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.395990 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.396061 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.396119 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.396462 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.396503 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":68,"threads_started":1}
I20260928 17:46:59.396770 8159 tablet_service.cc:1467] Processing CreateTablet for tablet f1d6f13da27245d2917075ecdc7cbfff (DEFAULT_TABLE table=test-table [id=e03a65022e814899880364e5d84917fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:59.397120 8159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d6f13da27245d2917075ecdc7cbfff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:59.397055 8292 tablet_service.cc:1467] Processing CreateTablet for tablet f1d6f13da27245d2917075ecdc7cbfff (DEFAULT_TABLE table=test-table [id=e03a65022e814899880364e5d84917fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:59.397348 8292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d6f13da27245d2917075ecdc7cbfff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:59.397281 8425 tablet_service.cc:1467] Processing CreateTablet for tablet f1d6f13da27245d2917075ecdc7cbfff (DEFAULT_TABLE table=test-table [id=e03a65022e814899880364e5d84917fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:46:59.397521 8425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d6f13da27245d2917075ecdc7cbfff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:59.400177 8505 tablet_bootstrap.cc:492] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: Bootstrap starting.
I20260928 17:46:59.400507 8506 tablet_bootstrap.cc:492] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: Bootstrap starting.
I20260928 17:46:59.400537 8507 tablet_bootstrap.cc:492] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13: Bootstrap starting.
I20260928 17:46:59.400812 8505 tablet_bootstrap.cc:654] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:59.401117 8507 tablet_bootstrap.cc:654] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:59.401117 8506 tablet_bootstrap.cc:654] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:59.401306 8505 log.cc:824] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:59.401670 8506 log.cc:824] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:59.401670 8507 log.cc:824] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:59.401979 8505 tablet_bootstrap.cc:492] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: No bootstrap required, opened a new log
I20260928 17:46:59.402048 8505 ts_tablet_manager.cc:1397] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:59.402312 8507 tablet_bootstrap.cc:492] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13: No bootstrap required, opened a new log
I20260928 17:46:59.402329 8506 tablet_bootstrap.cc:492] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: No bootstrap required, opened a new log
I20260928 17:46:59.402379 8506 ts_tablet_manager.cc:1397] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:59.402379 8507 ts_tablet_manager.cc:1397] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:46:59.403714 8505 raft_consensus.cc:348] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } }
I20260928 17:46:59.403841 8505 raft_consensus.cc:374] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:59.403872 8505 raft_consensus.cc:729] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8aeb6f301946bbbe5b7b08f23bd2d2, State: Initialized, Role: FOLLOWER
I20260928 17:46:59.403995 8505 consensus_queue.cc:260] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } }
I20260928 17:46:59.404249 8505 ts_tablet_manager.cc:1428] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:46:59.404256 8506 raft_consensus.cc:348] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } }
I20260928 17:46:59.404256 8507 raft_consensus.cc:348] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } }
I20260928 17:46:59.404412 8506 raft_consensus.cc:374] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:59.404412 8507 raft_consensus.cc:374] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:59.404452 8506 raft_consensus.cc:729] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9df138d9dd4cd5ad7d7aac822314ce, State: Initialized, Role: FOLLOWER
I20260928 17:46:59.404452 8507 raft_consensus.cc:729] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494e7a138f814af2832cff742b38ef13, State: Initialized, Role: FOLLOWER
I20260928 17:46:59.404563 8506 consensus_queue.cc:260] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } }
I20260928 17:46:59.404563 8507 consensus_queue.cc:260] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } }
I20260928 17:46:59.404839 8507 ts_tablet_manager.cc:1428] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:46:59.404830 8358 heartbeater.cc:503] Master 127.3.222.62:35251 was elected leader, sending a full tablet report...
I20260928 17:46:59.404872 8225 heartbeater.cc:503] Master 127.3.222.62:35251 was elected leader, sending a full tablet report...
I20260928 17:46:59.404966 8491 heartbeater.cc:503] Master 127.3.222.62:35251 was elected leader, sending a full tablet report...
I20260928 17:46:59.404825 8506 ts_tablet_manager.cc:1428] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 17:46:59.479257 8359 tablet.cc:2369] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:46:59.508453 8511 raft_consensus.cc:482] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:46:59.508572 8511 raft_consensus.cc:504] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } }
I20260928 17:46:59.508912 8511 leader_election.cc:288] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b9df138d9dd4cd5ad7d7aac822314ce (127.3.222.2:40117), 494e7a138f814af2832cff742b38ef13 (127.3.222.1:41241)
I20260928 17:46:59.512666 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.509139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:55641 (local address 127.3.222.2:40117)
0928 17:46:59.509180 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.509182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.509359 (+ 177us) server_negotiation.cc:407] Connection header received
0928 17:46:59.509384 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.509385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.509395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.509430 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:59.509567 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.510260 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.511728 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.512325 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.512383 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.512393 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.512558 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.512598 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:46:59.512666 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.509138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:53177 (local address 127.3.222.1:41241)
0928 17:46:59.509204 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.509206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.509216 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:46:59.509243 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.509244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.509253 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.509300 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:46:59.509490 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.510236 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.511728 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.512301 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.512382 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.512393 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.512558 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.512598 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 17:46:59.512671 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.509074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:41241 (local address 127.3.222.3:53177)
0928 17:46:59.509133 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.509144 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.509186 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.509323 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.509324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.509325 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:59.509412 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.509414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.510325 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.510326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.511623 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.511625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.512400 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.512401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.512483 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.512485 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.512539 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.512585 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":37}
I20260928 17:46:59.512686 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.509076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40117 (local address 127.3.222.3:55641)
0928 17:46:59.509264 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.509273 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.509324 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.509446 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.509447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.509448 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:46:59.509503 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.509504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.510326 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.510326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.511618 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:46:59.511621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.512400 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.512401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:46:59.512477 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.512482 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.512539 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.512584 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":67,"thread_start_us":90,"threads_started":1}
I20260928 17:46:59.513039 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d6f13da27245d2917075ecdc7cbfff" candidate_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494e7a138f814af2832cff742b38ef13" is_pre_election: true
I20260928 17:46:59.513051 8312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d6f13da27245d2917075ecdc7cbfff" candidate_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" is_pre_election: true
I20260928 17:46:59.513199 8312 raft_consensus.cc:2454] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af8aeb6f301946bbbe5b7b08f23bd2d2 in term 0.
I20260928 17:46:59.513199 8179 raft_consensus.cc:2454] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af8aeb6f301946bbbe5b7b08f23bd2d2 in term 0.
I20260928 17:46:59.513432 8379 leader_election.cc:302] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9df138d9dd4cd5ad7d7aac822314ce, af8aeb6f301946bbbe5b7b08f23bd2d2; no voters:
I20260928 17:46:59.513557 8511 raft_consensus.cc:2781] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:46:59.513608 8511 raft_consensus.cc:482] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:46:59.513636 8511 raft_consensus.cc:3037] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:59.514278 8511 raft_consensus.cc:504] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } }
I20260928 17:46:59.514398 8511 leader_election.cc:288] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [CANDIDATE]: Term 1 election: Requested vote from peers 1b9df138d9dd4cd5ad7d7aac822314ce (127.3.222.2:40117), 494e7a138f814af2832cff742b38ef13 (127.3.222.1:41241)
I20260928 17:46:59.514549 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d6f13da27245d2917075ecdc7cbfff" candidate_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494e7a138f814af2832cff742b38ef13"
I20260928 17:46:59.514587 8312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d6f13da27245d2917075ecdc7cbfff" candidate_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce"
I20260928 17:46:59.514622 8179 raft_consensus.cc:3037] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:59.514647 8312 raft_consensus.cc:3037] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:59.515303 8312 raft_consensus.cc:2454] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af8aeb6f301946bbbe5b7b08f23bd2d2 in term 1.
I20260928 17:46:59.515355 8179 raft_consensus.cc:2454] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af8aeb6f301946bbbe5b7b08f23bd2d2 in term 1.
I20260928 17:46:59.515450 8379 leader_election.cc:302] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9df138d9dd4cd5ad7d7aac822314ce, af8aeb6f301946bbbe5b7b08f23bd2d2; no voters:
I20260928 17:46:59.515529 8511 raft_consensus.cc:2781] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:59.515686 8511 raft_consensus.cc:686] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 1 LEADER]: Becoming Leader. State: Replica: af8aeb6f301946bbbe5b7b08f23bd2d2, State: Running, Role: LEADER
I20260928 17:46:59.515771 8511 consensus_queue.cc:237] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } }
I20260928 17:46:59.516626 8033 catalog_manager.cc:5495] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to af8aeb6f301946bbbe5b7b08f23bd2d2 (127.3.222.3). New cstate: current_term: 1 leader_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af8aeb6f301946bbbe5b7b08f23bd2d2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46167 } health_report { overall_health: HEALTHY } } }
I20260928 17:46:59.539655 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.537410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.3.222.3:46167)
0928 17:46:59.537471 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.537473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.537481 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:46:59.537513 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.537514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.537538 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.537611 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:46:59.537710 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.538160 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.538931 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:46:59.539124 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:46:59.539186 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:46:59.539350 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:46:59.539450 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:46:59.539511 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.539533 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.539550 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:46:59.541412 8312 raft_consensus.cc:1261] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [term 1 FOLLOWER]: Refusing update from remote peer af8aeb6f301946bbbe5b7b08f23bd2d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:59.541474 8179 raft_consensus.cc:1261] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [term 1 FOLLOWER]: Refusing update from remote peer af8aeb6f301946bbbe5b7b08f23bd2d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:46:59.541795 8511 consensus_queue.cc:1035] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9df138d9dd4cd5ad7d7aac822314ce" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:59.541929 8515 consensus_queue.cc:1035] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "494e7a138f814af2832cff742b38ef13" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:46:59.544219 8519 mvcc.cc:205] Tried to move back new op lower bound from 7334369769639157760 to 7334369769537155072. Current Snapshot: MvccSnapshot[applied={T|T < 7334369769639157760}]
I20260928 17:46:59.544803 8520 mvcc.cc:205] Tried to move back new op lower bound from 7334369769639157760 to 7334369769537155072. Current Snapshot: MvccSnapshot[applied={T|T < 7334369769639157760}]
I20260928 17:46:59.546073 8033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:59.546171 8033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:46:59.546892 8033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd4a0fe769e946c2a8f295f929d2253a: Sending DeleteTablet for 3 replicas of tablet f1d6f13da27245d2917075ecdc7cbfff
I20260928 17:46:59.547269 8292 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d6f13da27245d2917075ecdc7cbfff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:59 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:53184
I20260928 17:46:59.547266 8159 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d6f13da27245d2917075ecdc7cbfff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:59 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:47378
I20260928 17:46:59.547441 8528 tablet_replica.cc:330] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13: stopping tablet replica
I20260928 17:46:59.547478 8529 tablet_replica.cc:330] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: stopping tablet replica
I20260928 17:46:59.547467 8425 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d6f13da27245d2917075ecdc7cbfff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:46:59 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:36366
I20260928 17:46:59.547528 8528 raft_consensus.cc:2229] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:59.547549 8529 raft_consensus.cc:2229] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:46:59.547616 8528 raft_consensus.cc:2258] T f1d6f13da27245d2917075ecdc7cbfff P 494e7a138f814af2832cff742b38ef13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:59.547618 8529 raft_consensus.cc:2258] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:59.547736 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 8097
I20260928 17:46:59.548475 8529 ts_tablet_manager.cc:1905] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:59.550091 8529 ts_tablet_manager.cc:1918] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:46:59.550151 8529 log.cc:1196] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/f1d6f13da27245d2917075ecdc7cbfff
I20260928 17:46:59.550364 8529 ts_tablet_manager.cc:1939] T f1d6f13da27245d2917075ecdc7cbfff P 1b9df138d9dd4cd5ad7d7aac822314ce: Deleting consensus metadata
I20260928 17:46:59.550676 8020 catalog_manager.cc:4841] TS 1b9df138d9dd4cd5ad7d7aac822314ce (127.3.222.2:40117): tablet f1d6f13da27245d2917075ecdc7cbfff (table test-table [id=e03a65022e814899880364e5d84917fc]) successfully deleted
W20260928 17:46:59.554054 8021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:41241 (error 108)
I20260928 17:46:59.554282 8530 tablet_replica.cc:330] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: stopping tablet replica
I20260928 17:46:59.554376 8530 raft_consensus.cc:2229] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:46:59.554478 8530 raft_consensus.cc:2258] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:46:59.554526 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.554420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:41241 (local address 127.0.0.1:47394)
0928 17:46:59.554464 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.554493 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:41241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 17:46:59.554637 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 8230
W20260928 17:46:59.554653 8021 catalog_manager.cc:4584] TS 494e7a138f814af2832cff742b38ef13 (127.3.222.1:41241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1d6f13da27245d2917075ecdc7cbfff: Network error: Client connection negotiation failed: client connection to 127.3.222.1:41241: connect: Connection refused (error 111)
I20260928 17:46:59.555332 8530 ts_tablet_manager.cc:1905] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:46:59.557159 8530 ts_tablet_manager.cc:1918] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:46:59.557219 8530 log.cc:1196] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/f1d6f13da27245d2917075ecdc7cbfff
I20260928 17:46:59.557457 8530 ts_tablet_manager.cc:1939] T f1d6f13da27245d2917075ecdc7cbfff P af8aeb6f301946bbbe5b7b08f23bd2d2: Deleting consensus metadata
I20260928 17:46:59.557982 8019 catalog_manager.cc:4841] TS af8aeb6f301946bbbe5b7b08f23bd2d2 (127.3.222.3:46167): tablet f1d6f13da27245d2917075ecdc7cbfff (table test-table [id=e03a65022e814899880364e5d84917fc]) successfully deleted
I20260928 17:46:59.559721 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 8363
I20260928 17:46:59.563758 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 8004
2026-09-28T17:46:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1858 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T17:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:46:59Z Disabled control of system clock
I20260928 17:46:59.589089 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35661
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:35661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 17:46:59.666390 8535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:59.666577 8535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:59.666596 8535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:59.667737 8535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:46:59.667778 8535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:59.667790 8535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:59.667801 8535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:46:59.667814 8535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:46:59.669135 8535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:35661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:59.669364 8535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:59.669627 8535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:59.674059 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:59.674122 8535 server_base.cc:1034] running on GCE node
W20260928 17:46:59.674156 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.674072 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:59.674526 8535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:59.674772 8535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:59.675920 8535 hybrid_clock.cc:648] HybridClock initialized: now 1790617619675905 us; error 27 us; skew 500 ppm
I20260928 17:46:59.677027 8535 webserver.cc:460] Webserver started at http://127.3.222.62:46727/ using document root <none> and password file <none>
I20260928 17:46:59.677207 8535 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:59.677245 8535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:59.677330 8535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:59.678288 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "040d9cf7fd2546f5a18ddcd20c30a257"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "e12d69f3980ef4e5d88b108c68935676"
server_key_iv: "b0008968af40a02f82841c953dfa3b1e"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.678565 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "040d9cf7fd2546f5a18ddcd20c30a257"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "e12d69f3980ef4e5d88b108c68935676"
server_key_iv: "b0008968af40a02f82841c953dfa3b1e"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.679698 8535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:59.680284 8549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.680459 8535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:59.680544 8535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "040d9cf7fd2546f5a18ddcd20c30a257"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "e12d69f3980ef4e5d88b108c68935676"
server_key_iv: "b0008968af40a02f82841c953dfa3b1e"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.680614 8535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:59.683809 8535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:59.684033 8535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:59.684167 8535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:59.687666 8535 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:35661
I20260928 17:46:59.687683 8601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:35661 every 8 connection(s)
I20260928 17:46:59.687978 8535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:46:59.688578 8602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:46:59.691115 8602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257: Bootstrap starting.
I20260928 17:46:59.691664 8602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257: Neither blocks nor log segments found. Creating new log.
I20260928 17:46:59.691921 8602 log.cc:824] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257: Log is configured to *not* fsync() on all Append() calls
I20260928 17:46:59.692551 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 8535
I20260928 17:46:59.692588 8602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257: No bootstrap required, opened a new log
I20260928 17:46:59.692653 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:46:59.692860 3960 external_mini_cluster.cc:1428] Setting key cb0743d9b224decff2a13aa642b97c5c
I20260928 17:46:59.694403 8602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040d9cf7fd2546f5a18ddcd20c30a257" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35661 } }
I20260928 17:46:59.694535 8602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:46:59.694567 8602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040d9cf7fd2546f5a18ddcd20c30a257, State: Initialized, Role: FOLLOWER
I20260928 17:46:59.694730 8602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040d9cf7fd2546f5a18ddcd20c30a257" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35661 } }
I20260928 17:46:59.694813 8602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:46:59.694845 8602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:46:59.694902 8602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:46:59.695523 8602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040d9cf7fd2546f5a18ddcd20c30a257" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35661 } }
I20260928 17:46:59.695603 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.693323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.3.222.62:35661)
0928 17:46:59.693497 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.693504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.693527 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:46:59.694372 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.694377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.694420 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.694525 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:59.695167 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:59.695170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:59.695171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:46:59.695175 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:59.695239 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:46:59.695255 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:59.695329 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.695367 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.695422 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260928 17:46:59.695621 8602 leader_election.cc:302] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 040d9cf7fd2546f5a18ddcd20c30a257; no voters:
I20260928 17:46:59.695755 8602 leader_election.cc:288] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:46:59.696026 8607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:46:59.696132 8607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [term 1 LEADER]: Becoming Leader. State: Replica: 040d9cf7fd2546f5a18ddcd20c30a257, State: Running, Role: LEADER
I20260928 17:46:59.696205 8607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040d9cf7fd2546f5a18ddcd20c30a257" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35661 } }
I20260928 17:46:59.696380 8602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:46:59.696522 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "040d9cf7fd2546f5a18ddcd20c30a257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040d9cf7fd2546f5a18ddcd20c30a257" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35661 } } }
I20260928 17:46:59.696537 8609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040d9cf7fd2546f5a18ddcd20c30a257. Latest consensus state: current_term: 1 leader_uuid: "040d9cf7fd2546f5a18ddcd20c30a257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040d9cf7fd2546f5a18ddcd20c30a257" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35661 } } }
I20260928 17:46:59.696669 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:59.696688 8609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257 [sys.catalog]: This master's current role is: LEADER
I20260928 17:46:59.697118 8612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:46:59.697569 8612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:46:59.698931 8612 catalog_manager.cc:1347] Generated new cluster ID: 8db7db6dc57647329e7ab3226afe9ea2
I20260928 17:46:59.698975 8612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:46:59.707803 8612 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:46:59.707952 8612 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:46:59.717252 8612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257: Generated new TSK 0
I20260928 17:46:59.717481 8612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:46:59.725708 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35661
--builtin_ntp_servers=127.3.222.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 17:46:59.808306 8626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:59.808507 8626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:59.808537 8626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:59.809690 8626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:59.809726 8626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:59.809760 8626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:46:59.811117 8626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:59.811364 8626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:59.811620 8626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:59.817330 8631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.817373 8632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.817379 8634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:59.817694 8626 server_base.cc:1034] running on GCE node
I20260928 17:46:59.817940 8626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:59.818169 8626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:59.819291 8626 hybrid_clock.cc:648] HybridClock initialized: now 1790617619819268 us; error 36 us; skew 500 ppm
I20260928 17:46:59.820524 8626 webserver.cc:460] Webserver started at http://127.3.222.1:36789/ using document root <none> and password file <none>
I20260928 17:46:59.820731 8626 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:59.820778 8626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:59.820906 8626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:59.821936 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "e8c34a0b892546c9a414e8a034328cd8"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "1bc2321adfc5178a093e1777b2b1ca83"
server_key_iv: "b3942e314e39a53dbaaef1f73622a03b"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.822260 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "e8c34a0b892546c9a414e8a034328cd8"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "1bc2321adfc5178a093e1777b2b1ca83"
server_key_iv: "b3942e314e39a53dbaaef1f73622a03b"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.823485 8626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:46:59.824126 8640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.824286 8626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:59.824381 8626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "e8c34a0b892546c9a414e8a034328cd8"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "1bc2321adfc5178a093e1777b2b1ca83"
server_key_iv: "b3942e314e39a53dbaaef1f73622a03b"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.824448 8626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:59.827589 8626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:59.827828 8626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:59.827943 8626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:59.828136 8626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:59.828409 8626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:59.828440 8626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.828465 8626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:59.828482 8626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.833657 8626 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:41937
I20260928 17:46:59.833709 8753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:41937 every 8 connection(s)
I20260928 17:46:59.833992 8626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:46:59.836632 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.834604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:46955 (local address 127.3.222.62:35661)
0928 17:46:59.834645 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.834647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.834842 (+ 195us) server_negotiation.cc:407] Connection header received
0928 17:46:59.835298 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.835300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.835308 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.835383 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:59.836444 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:59.836446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:59.836446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:46:59.836450 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:59.836468 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:46:59.836473 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:59.836518 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.836567 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.836580 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:46:59.836790 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.834581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35661 (local address 127.3.222.1:46955)
0928 17:46:59.834733 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.834763 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.835182 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.835440 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.835443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.835450 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:59.836273 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:46:59.836273 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:59.836299 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:46:59.836343 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:59.836520 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:59.836522 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:59.836524 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.836568 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.836612 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260928 17:46:59.837222 8754 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35661
I20260928 17:46:59.837296 8754 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:59.837430 8754 heartbeater.cc:511] Master 127.3.222.62:35661 requested a full tablet report, sending...
I20260928 17:46:59.837881 8566 ts_manager.cc:194] Registered new tserver with Master: e8c34a0b892546c9a414e8a034328cd8 (127.3.222.1:41937)
I20260928 17:46:59.840512 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 8626
I20260928 17:46:59.840592 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:46:59.840742 3960 external_mini_cluster.cc:1428] Setting key 31e81830f5ef3da023143d5d989be0a9
I20260928 17:46:59.841535 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35661
--builtin_ntp_servers=127.3.222.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 17:46:59.918943 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:46:59.919139 8757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:46:59.919167 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:46:59.920317 8757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:46:59.920354 8757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:46:59.920388 8757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:46:59.921659 8757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:46:59.921875 8757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:46:59.922128 8757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:46:59.926913 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.926923 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:46:59.926923 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:46:59.927069 8757 server_base.cc:1034] running on GCE node
I20260928 17:46:59.927382 8757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:46:59.927634 8757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:46:59.928767 8757 hybrid_clock.cc:648] HybridClock initialized: now 1790617619928778 us; error 52 us; skew 500 ppm
I20260928 17:46:59.929917 8757 webserver.cc:460] Webserver started at http://127.3.222.2:35245/ using document root <none> and password file <none>
I20260928 17:46:59.930109 8757 fs_manager.cc:359] Metadata directory not provided
I20260928 17:46:59.930150 8757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:46:59.930281 8757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:46:59.931313 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "8c9828a3da9b427ba95217587d3ec11c"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "b0a5274e0e20a47091ec166793e4fd89"
server_key_iv: "4365f3bacb1be2b866e90c6bce9b8249"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.931623 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "8c9828a3da9b427ba95217587d3ec11c"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "b0a5274e0e20a47091ec166793e4fd89"
server_key_iv: "4365f3bacb1be2b866e90c6bce9b8249"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.932789 8757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:46:59.933389 8771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.933543 8757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:46:59.933619 8757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "8c9828a3da9b427ba95217587d3ec11c"
format_stamp: "Formatted at 2026-09-28 17:46:59 on dist-test-slave-w3bm"
server_key: "b0a5274e0e20a47091ec166793e4fd89"
server_key_iv: "4365f3bacb1be2b866e90c6bce9b8249"
server_key_version: "encryptionkey@0"
I20260928 17:46:59.933677 8757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:46:59.936805 8757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:46:59.937026 8757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:46:59.937132 8757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:46:59.937311 8757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:46:59.937587 8757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:46:59.937618 8757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.937644 8757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:46:59.937668 8757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:46:59.942839 8757 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:40641
I20260928 17:46:59.942890 8884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:40641 every 8 connection(s)
I20260928 17:46:59.943161 8757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:46:59.945497 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 8757
I20260928 17:46:59.945585 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:46:59.945739 3960 external_mini_cluster.cc:1428] Setting key 9a8f0d64240a8e5abbc63c4db9ced7a3
I20260928 17:46:59.946036 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.943820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:49053 (local address 127.3.222.62:35661)
0928 17:46:59.943891 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:46:59.943894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:46:59.944085 (+ 191us) server_negotiation.cc:407] Connection header received
0928 17:46:59.944547 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:46:59.944549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:46:59.944564 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:46:59.944614 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:46:59.945703 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:46:59.945705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:46:59.945705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:46:59.945710 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:46:59.945727 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:46:59.945731 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:46:59.945781 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:46:59.945948 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 17:46:59.945965 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:46:59.946199 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:46:59.943787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35661 (local address 127.3.222.2:49053)
0928 17:46:59.943974 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 17:46:59.944006 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:46:59.944454 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:46:59.944683 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:46:59.944687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:46:59.944697 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:46:59.945542 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:46:59.945542 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:46:59.945569 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:46:59.945601 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:46:59.945832 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:46:59.945837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:46:59.945841 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:46:59.945936 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 17:46:59.946000 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260928 17:46:59.946595 8885 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35661
I20260928 17:46:59.946678 8885 heartbeater.cc:464] Registering TS with master...
I20260928 17:46:59.946728 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35661
--builtin_ntp_servers=127.3.222.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 17:46:59.946861 8885 heartbeater.cc:511] Master 127.3.222.62:35661 requested a full tablet report, sending...
I20260928 17:46:59.947239 8566 ts_manager.cc:194] Registered new tserver with Master: 8c9828a3da9b427ba95217587d3ec11c (127.3.222.2:40641)
W20260928 17:47:00.023170 8888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:00.023399 8888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:00.023420 8888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:00.024514 8888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:00.024546 8888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:00.024570 8888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:00.025802 8888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:00.026031 8888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:00.026255 8888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:00.030932 8896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:00.031067 8894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:00.031131 8893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:00.031138 8888 server_base.cc:1034] running on GCE node
I20260928 17:47:00.031433 8888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:00.031622 8888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:00.032801 8888 hybrid_clock.cc:648] HybridClock initialized: now 1790617620032783 us; error 36 us; skew 500 ppm
I20260928 17:47:00.034049 8888 webserver.cc:460] Webserver started at http://127.3.222.3:40895/ using document root <none> and password file <none>
I20260928 17:47:00.034251 8888 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:00.034297 8888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:00.034433 8888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:00.035498 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "6633248a1bba43208d5c057df6e98b5b"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "5ff67557d0faaa231e7f2abb28990731"
server_key_iv: "a2997be5303cb84c41716b2b6c78cc40"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.035818 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "6633248a1bba43208d5c057df6e98b5b"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "5ff67557d0faaa231e7f2abb28990731"
server_key_iv: "a2997be5303cb84c41716b2b6c78cc40"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.037149 8888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 17:47:00.037848 8902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:00.037995 8888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:00.038074 8888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "6633248a1bba43208d5c057df6e98b5b"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "5ff67557d0faaa231e7f2abb28990731"
server_key_iv: "a2997be5303cb84c41716b2b6c78cc40"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.038134 8888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:00.041410 8888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:00.041637 8888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:00.041755 8888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:00.041936 8888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:00.042228 8888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:00.042260 8888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:00.042285 8888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:00.042302 8888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:00.047435 8888 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:36579
I20260928 17:47:00.047470 9015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:36579 every 8 connection(s)
I20260928 17:47:00.047751 8888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:47:00.050431 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.048381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:44001 (local address 127.3.222.62:35661)
0928 17:47:00.048442 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.048444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.048635 (+ 191us) server_negotiation.cc:407] Connection header received
0928 17:47:00.049131 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.049132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.049140 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.049190 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.050198 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.050200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.050200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.050204 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.050219 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.050223 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.050272 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.050347 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.050362 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:47:00.050603 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.048350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35661 (local address 127.3.222.3:44001)
0928 17:47:00.048524 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.048554 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:00.049032 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:00.049253 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:00.049256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:00.049264 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:00.050064 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:00.050064 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:00.050091 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:00.050116 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:00.050279 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:00.050283 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:00.050286 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:00.050349 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 17:47:00.050409 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 17:47:00.050885 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 8888
I20260928 17:47:00.050977 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:47:00.051067 9016 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35661
I20260928 17:47:00.051139 3960 external_mini_cluster.cc:1428] Setting key 75dc5f7dfad080093455009102b32d1b
I20260928 17:47:00.051151 9016 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:00.051302 9016 heartbeater.cc:511] Master 127.3.222.62:35661 requested a full tablet report, sending...
I20260928 17:47:00.051638 8566 ts_manager.cc:194] Registered new tserver with Master: 6633248a1bba43208d5c057df6e98b5b (127.3.222.3:36579)
I20260928 17:47:00.052124 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:47:00.054087 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.053315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.3.222.62:35661)
0928 17:47:00.053373 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.053375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.053446 (+ 71us) server_negotiation.cc:407] Connection header received
0928 17:47:00.053482 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.053483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.053487 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.053529 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.053931 (+ 402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.053932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.053933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.053936 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.053949 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.053952 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.053989 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.054017 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.054028 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:47:00.055322 8566 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:47:00.056845 8566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:47:00.061049 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.059591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40641 (local address 127.0.0.1:41854)
0928 17:47:00.059950 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.059987 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:00.060038 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:00.060183 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:00.060190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:00.060197 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:00.060640 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:00.060641 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:00.060657 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:00.060680 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:00.060902 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:00.060908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:00.060911 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:00.060965 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:47:00.060978 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":80,"threads_started":1}
I20260928 17:47:00.061070 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.059795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.3.222.2:40641)
0928 17:47:00.060020 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.060023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.060031 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:00.060090 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.060091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.060099 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.060161 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.060780 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.060786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.060786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.060791 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.060829 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.060856 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.060908 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.060982 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.060996 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260928 17:47:00.061379 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.059612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:36579 (local address 127.0.0.1:47094)
0928 17:47:00.060201 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.060209 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:00.060272 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:00.060632 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:00.060633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:00.060634 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:00.061026 (+ 392us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:00.061026 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:00.061033 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:00.061047 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:00.061260 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:00.061261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:00.061262 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:47:00.061304 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 17:47:00.061316 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":81,"threads_started":1}
I20260928 17:47:00.061405 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.059810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.3.222.3:36579)
0928 17:47:00.059960 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.059964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.060253 (+ 289us) server_negotiation.cc:407] Connection header received
0928 17:47:00.060331 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.060333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.060341 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.060401 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.061128 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.061134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.061134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.061138 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.061170 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.061189 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.061237 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.061314 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.061328 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260928 17:47:00.061888 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.059591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:41937 (local address 127.0.0.1:57984)
0928 17:47:00.060097 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.060104 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:00.060134 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:00.060627 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:00.060628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:00.060631 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:00.061520 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:00.061521 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:00.061527 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:00.061538 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:00.061760 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:00.061761 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:00.061762 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:47:00.061817 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 17:47:00.061828 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":1,"thread_start_us":48,"threads_started":1}
I20260928 17:47:00.061914 9027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.059774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.3.222.1:41937)
0928 17:47:00.059938 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.059942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.060157 (+ 215us) server_negotiation.cc:407] Connection header received
0928 17:47:00.060185 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.060188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.060198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.060267 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.061616 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.061625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.061625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.061629 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.061664 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.061691 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.061747 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.061823 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.061837 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 17:47:00.062060 8819 tablet_service.cc:1467] Processing CreateTablet for tablet 6e66094af1a944ca90be87093cefabdc (DEFAULT_TABLE table=test-table [id=b00821f46f0d4295a249872514c2b978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:00.062052 8950 tablet_service.cc:1467] Processing CreateTablet for tablet 6e66094af1a944ca90be87093cefabdc (DEFAULT_TABLE table=test-table [id=b00821f46f0d4295a249872514c2b978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:00.062383 8950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e66094af1a944ca90be87093cefabdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:00.062390 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e66094af1a944ca90be87093cefabdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:00.062774 8688 tablet_service.cc:1467] Processing CreateTablet for tablet 6e66094af1a944ca90be87093cefabdc (DEFAULT_TABLE table=test-table [id=b00821f46f0d4295a249872514c2b978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:00.063057 8688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e66094af1a944ca90be87093cefabdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:00.064801 9030 tablet_bootstrap.cc:492] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: Bootstrap starting.
I20260928 17:47:00.065407 9030 tablet_bootstrap.cc:654] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:00.065513 9031 tablet_bootstrap.cc:492] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: Bootstrap starting.
I20260928 17:47:00.065805 9030 log.cc:824] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:00.065972 9032 tablet_bootstrap.cc:492] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8: Bootstrap starting.
I20260928 17:47:00.066116 9031 tablet_bootstrap.cc:654] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:00.066365 9031 log.cc:824] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:00.066573 9032 tablet_bootstrap.cc:654] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:00.066759 9030 tablet_bootstrap.cc:492] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: No bootstrap required, opened a new log
I20260928 17:47:00.066828 9030 ts_tablet_manager.cc:1397] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:00.066867 9032 log.cc:824] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:00.067046 9031 tablet_bootstrap.cc:492] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: No bootstrap required, opened a new log
I20260928 17:47:00.067107 9031 ts_tablet_manager.cc:1397] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:00.067492 9032 tablet_bootstrap.cc:492] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8: No bootstrap required, opened a new log
I20260928 17:47:00.067739 9032 ts_tablet_manager.cc:1397] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:00.069286 9030 raft_consensus.cc:348] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } }
I20260928 17:47:00.069456 9030 raft_consensus.cc:374] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:00.069489 9030 raft_consensus.cc:729] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9828a3da9b427ba95217587d3ec11c, State: Initialized, Role: FOLLOWER
I20260928 17:47:00.069410 9031 raft_consensus.cc:348] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } }
I20260928 17:47:00.069528 9031 raft_consensus.cc:374] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:00.069442 9032 raft_consensus.cc:348] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } }
I20260928 17:47:00.069555 9031 raft_consensus.cc:729] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6633248a1bba43208d5c057df6e98b5b, State: Initialized, Role: FOLLOWER
I20260928 17:47:00.069583 9032 raft_consensus.cc:374] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:00.069619 9032 raft_consensus.cc:729] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c34a0b892546c9a414e8a034328cd8, State: Initialized, Role: FOLLOWER
I20260928 17:47:00.069620 9030 consensus_queue.cc:260] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } }
I20260928 17:47:00.069653 9031 consensus_queue.cc:260] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } }
I20260928 17:47:00.069743 9032 consensus_queue.cc:260] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } }
I20260928 17:47:00.069866 9030 ts_tablet_manager.cc:1428] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:47:00.069869 9031 ts_tablet_manager.cc:1428] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:47:00.069974 8885 heartbeater.cc:503] Master 127.3.222.62:35661 was elected leader, sending a full tablet report...
I20260928 17:47:00.069996 9032 ts_tablet_manager.cc:1428] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:47:00.070109 8754 heartbeater.cc:503] Master 127.3.222.62:35661 was elected leader, sending a full tablet report...
I20260928 17:47:00.070199 9016 heartbeater.cc:503] Master 127.3.222.62:35661 was elected leader, sending a full tablet report...
W20260928 17:47:00.084442 8755 tablet.cc:2369] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:47:00.127117 9038 raft_consensus.cc:482] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:00.127386 9038 raft_consensus.cc:504] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } }
I20260928 17:47:00.127755 9038 leader_election.cc:288] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c9828a3da9b427ba95217587d3ec11c (127.3.222.2:40641), 6633248a1bba43208d5c057df6e98b5b (127.3.222.3:36579)
I20260928 17:47:00.129045 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.127933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40641 (local address 127.3.222.1:43599)
0928 17:47:00.128016 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.128028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:00.128067 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:00.128209 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:00.128211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:00.128213 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:00.128622 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:00.128623 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:00.128629 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:00.128639 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:00.128925 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:00.128925 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:00.128926 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:47:00.128971 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 17:47:00.128980 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 17:47:00.129055 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.127937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:36579 (local address 127.3.222.1:49625)
0928 17:47:00.128201 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.128219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:00.128265 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:00.128389 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:00.128390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:00.128392 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:00.128743 (+ 351us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:00.128743 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:00.128748 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:00.128757 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:00.128906 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:00.128906 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:00.128907 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:47:00.128956 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 17:47:00.128970 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":105,"thread_start_us":104,"threads_started":1}
I20260928 17:47:00.129068 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.128012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:43599 (local address 127.3.222.2:40641)
0928 17:47:00.128056 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.128058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.128068 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:00.128119 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.128120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.128125 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.128184 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.128731 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.128733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.128733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.128737 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.128753 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.128757 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.128806 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.128991 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.129003 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:47:00.129068 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.128000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:49625 (local address 127.3.222.3:36579)
0928 17:47:00.128064 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.128066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.128286 (+ 220us) server_negotiation.cc:407] Connection header received
0928 17:47:00.128309 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.128310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.128313 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.128358 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.128822 (+ 464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.128824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.128824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.128828 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.128843 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.128847 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.128902 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.128983 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.129000 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:47:00.129359 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e66094af1a944ca90be87093cefabdc" candidate_uuid: "e8c34a0b892546c9a414e8a034328cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9828a3da9b427ba95217587d3ec11c" is_pre_election: true
I20260928 17:47:00.129400 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e66094af1a944ca90be87093cefabdc" candidate_uuid: "e8c34a0b892546c9a414e8a034328cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6633248a1bba43208d5c057df6e98b5b" is_pre_election: true
I20260928 17:47:00.129513 8839 raft_consensus.cc:2454] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8c34a0b892546c9a414e8a034328cd8 in term 0.
I20260928 17:47:00.129536 8970 raft_consensus.cc:2454] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8c34a0b892546c9a414e8a034328cd8 in term 0.
I20260928 17:47:00.129726 8641 leader_election.cc:302] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9828a3da9b427ba95217587d3ec11c, e8c34a0b892546c9a414e8a034328cd8; no voters:
I20260928 17:47:00.129865 9038 raft_consensus.cc:2781] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:47:00.129927 9038 raft_consensus.cc:482] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:47:00.129943 9038 raft_consensus.cc:3037] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:00.130630 9038 raft_consensus.cc:504] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } }
I20260928 17:47:00.130743 9038 leader_election.cc:288] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [CANDIDATE]: Term 1 election: Requested vote from peers 8c9828a3da9b427ba95217587d3ec11c (127.3.222.2:40641), 6633248a1bba43208d5c057df6e98b5b (127.3.222.3:36579)
I20260928 17:47:00.130878 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e66094af1a944ca90be87093cefabdc" candidate_uuid: "e8c34a0b892546c9a414e8a034328cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9828a3da9b427ba95217587d3ec11c"
I20260928 17:47:00.130877 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e66094af1a944ca90be87093cefabdc" candidate_uuid: "e8c34a0b892546c9a414e8a034328cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6633248a1bba43208d5c057df6e98b5b"
I20260928 17:47:00.130954 8839 raft_consensus.cc:3037] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:00.130956 8970 raft_consensus.cc:3037] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:00.131618 8839 raft_consensus.cc:2454] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8c34a0b892546c9a414e8a034328cd8 in term 1.
I20260928 17:47:00.131717 8970 raft_consensus.cc:2454] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8c34a0b892546c9a414e8a034328cd8 in term 1.
I20260928 17:47:00.131762 8641 leader_election.cc:302] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9828a3da9b427ba95217587d3ec11c, e8c34a0b892546c9a414e8a034328cd8; no voters:
I20260928 17:47:00.131877 9038 raft_consensus.cc:2781] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:00.132010 9038 raft_consensus.cc:686] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [term 1 LEADER]: Becoming Leader. State: Replica: e8c34a0b892546c9a414e8a034328cd8, State: Running, Role: LEADER
I20260928 17:47:00.132083 9038 consensus_queue.cc:237] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } }
I20260928 17:47:00.132867 8564 catalog_manager.cc:5495] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8c34a0b892546c9a414e8a034328cd8 (127.3.222.1). New cstate: current_term: 1 leader_uuid: "e8c34a0b892546c9a414e8a034328cd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8c34a0b892546c9a414e8a034328cd8" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 } health_report { overall_health: UNKNOWN } } }
I20260928 17:47:00.137190 9027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.136393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.3.222.1:41937)
0928 17:47:00.136432 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.136433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.136456 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:47:00.136522 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.136522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.136532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.136575 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.136975 (+ 400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.136976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.136977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.136981 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.136997 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.137020 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.137071 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.137121 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.137137 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:47:00.138695 8839 raft_consensus.cc:1261] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [term 1 FOLLOWER]: Refusing update from remote peer e8c34a0b892546c9a414e8a034328cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:00.138963 8970 raft_consensus.cc:1261] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [term 1 FOLLOWER]: Refusing update from remote peer e8c34a0b892546c9a414e8a034328cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:00.139000 9040 consensus_queue.cc:1035] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9828a3da9b427ba95217587d3ec11c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:00.139173 9040 consensus_queue.cc:1035] T 6e66094af1a944ca90be87093cefabdc P e8c34a0b892546c9a414e8a034328cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6633248a1bba43208d5c057df6e98b5b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 36579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:00.141150 9043 mvcc.cc:205] Tried to move back new op lower bound from 7334369772085706752 to 7334369772061364224. Current Snapshot: MvccSnapshot[applied={T|T < 7334369772085706752}]
I20260928 17:47:00.141841 9045 mvcc.cc:205] Tried to move back new op lower bound from 7334369772085706752 to 7334369772061364224. Current Snapshot: MvccSnapshot[applied={T|T < 7334369772085706752}]
I20260928 17:47:00.143829 8564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:00.143925 8564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:00.144542 8564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 040d9cf7fd2546f5a18ddcd20c30a257: Sending DeleteTablet for 3 replicas of tablet 6e66094af1a944ca90be87093cefabdc
I20260928 17:47:00.144847 8819 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e66094af1a944ca90be87093cefabdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:00 UTC) from {username='slave'} at 127.0.0.1:41854
I20260928 17:47:00.144847 8688 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e66094af1a944ca90be87093cefabdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:00 UTC) from {username='slave'} at 127.0.0.1:57984
I20260928 17:47:00.144866 8950 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e66094af1a944ca90be87093cefabdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:00 UTC) from {username='slave'} at 127.0.0.1:47094
I20260928 17:47:00.145052 9054 tablet_replica.cc:330] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: stopping tablet replica
I20260928 17:47:00.145061 9053 tablet_replica.cc:330] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: stopping tablet replica
I20260928 17:47:00.145145 9054 raft_consensus.cc:2229] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:00.145241 9054 raft_consensus.cc:2258] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:00.145255 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 8626
I20260928 17:47:00.145640 9054 ts_tablet_manager.cc:1905] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:00.145251 9053 raft_consensus.cc:2229] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:00.145919 9053 raft_consensus.cc:2258] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:00.146292 9053 ts_tablet_manager.cc:1905] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:00.147349 9054 ts_tablet_manager.cc:1918] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:00.147413 9054 log.cc:1196] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/6e66094af1a944ca90be87093cefabdc
I20260928 17:47:00.147642 9054 ts_tablet_manager.cc:1939] T 6e66094af1a944ca90be87093cefabdc P 6633248a1bba43208d5c057df6e98b5b: Deleting consensus metadata
I20260928 17:47:00.147734 9053 ts_tablet_manager.cc:1918] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:00.147799 9053 log.cc:1196] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/6e66094af1a944ca90be87093cefabdc
I20260928 17:47:00.148011 9053 ts_tablet_manager.cc:1939] T 6e66094af1a944ca90be87093cefabdc P 8c9828a3da9b427ba95217587d3ec11c: Deleting consensus metadata
I20260928 17:47:00.148053 8553 catalog_manager.cc:4841] TS 6633248a1bba43208d5c057df6e98b5b (127.3.222.3:36579): tablet 6e66094af1a944ca90be87093cefabdc (table test-table [id=b00821f46f0d4295a249872514c2b978]) successfully deleted
I20260928 17:47:00.148485 8550 catalog_manager.cc:4841] TS 8c9828a3da9b427ba95217587d3ec11c (127.3.222.2:40641): tablet 6e66094af1a944ca90be87093cefabdc (table test-table [id=b00821f46f0d4295a249872514c2b978]) successfully deleted
W20260928 17:47:00.150704 8552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:41937 (error 108)
I20260928 17:47:00.150991 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 8757
I20260928 17:47:00.151157 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.151040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:41937 (local address 127.0.0.1:57990)
0928 17:47:00.151092 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.151129 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:41937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 17:47:00.151603 8552 catalog_manager.cc:4584] TS e8c34a0b892546c9a414e8a034328cd8 (127.3.222.1:41937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e66094af1a944ca90be87093cefabdc: Network error: Client connection negotiation failed: client connection to 127.3.222.1:41937: connect: Connection refused (error 111)
I20260928 17:47:00.155401 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 8888
I20260928 17:47:00.158874 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 8535
2026-09-28T17:47:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (594 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 17:47:00.176725 3960 mini_oidc.cc:152] Starting JWKS server
I20260928 17:47:00.177743 3960 webserver.cc:460] Webserver started at https://127.0.0.1:38579/ using document root <none> and password file <none>
I20260928 17:47:00.177845 3960 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 17:47:00.178001 3960 webserver.cc:460] Webserver started at http://127.0.0.1:42505/ using document root <none> and password file <none>
2026-09-28T17:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:00Z Disabled control of system clock
I20260928 17:47:00.185531 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36137
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:36137
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38579/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/testchainca.cert with env {}
W20260928 17:47:00.264134 9062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:00.264333 9062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:00.264353 9062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:00.265826 9062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:00.265869 9062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:00.265882 9062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:00.265892 9062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:00.265902 9062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 17:47:00.265914 9062 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 17:47:00.265938 9062 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38579/jwks.json
I20260928 17:47:00.267484 9062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:36137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38579/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:00.267707 9062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:00.267895 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:00.272571 9067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:00.272580 9068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:00.272648 9070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:00.272858 9062 server_base.cc:1034] running on GCE node
I20260928 17:47:00.273044 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:00.273258 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:00.274402 9062 hybrid_clock.cc:648] HybridClock initialized: now 1790617620274385 us; error 25 us; skew 500 ppm
I20260928 17:47:00.275545 9062 webserver.cc:460] Webserver started at http://127.3.222.62:38051/ using document root <none> and password file <none>
I20260928 17:47:00.275744 9062 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:00.275791 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:00.275908 9062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:00.276852 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "a2787a6583a8485d9270dbc40d793edb"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "ec8e9b2a6ea731b820b9b9428a1c1490"
server_key_iv: "286140a713b73d3f72e587ad2a67961d"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.277156 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "a2787a6583a8485d9270dbc40d793edb"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "ec8e9b2a6ea731b820b9b9428a1c1490"
server_key_iv: "286140a713b73d3f72e587ad2a67961d"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.278281 9062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:47:00.278872 9076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:00.279030 9062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:00.279107 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "a2787a6583a8485d9270dbc40d793edb"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "ec8e9b2a6ea731b820b9b9428a1c1490"
server_key_iv: "286140a713b73d3f72e587ad2a67961d"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.279167 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:00.323889 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:00.324196 9062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:00.324321 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:00.327945 9062 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:36137
I20260928 17:47:00.327987 9128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:36137 every 8 connection(s)
I20260928 17:47:00.328290 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:00.329169 9129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:00.329644 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 9062
I20260928 17:47:00.329727 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:00.329895 3960 external_mini_cluster.cc:1428] Setting key c6a4b100448d1b920a939368a0363eba
I20260928 17:47:00.331969 9129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb: Bootstrap starting.
I20260928 17:47:00.332639 9129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:00.332911 9129 log.cc:824] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:00.333539 9129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb: No bootstrap required, opened a new log
I20260928 17:47:00.335582 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.330368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.3.222.62:36137)
0928 17:47:00.330566 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.330575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.330607 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:47:00.331096 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.331111 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.331192 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.331352 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.331553 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.332840 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.333604 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.334198 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.334330 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.334868 (+ 538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.334872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.334872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.335076 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.335122 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.335137 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.335200 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.335292 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.335378 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260928 17:47:00.335716 9129 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2787a6583a8485d9270dbc40d793edb" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36137 } }
I20260928 17:47:00.335844 9129 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:00.335870 9129 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2787a6583a8485d9270dbc40d793edb, State: Initialized, Role: FOLLOWER
I20260928 17:47:00.335980 9129 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2787a6583a8485d9270dbc40d793edb" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36137 } }
I20260928 17:47:00.336051 9129 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:00.336088 9129 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:00.336129 9129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:00.336441 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:47:00.336714 9129 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2787a6583a8485d9270dbc40d793edb" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36137 } }
I20260928 17:47:00.336812 9129 leader_election.cc:302] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2787a6583a8485d9270dbc40d793edb; no voters:
I20260928 17:47:00.336953 9129 leader_election.cc:288] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:00.336990 9134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:00.337128 9134 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [term 1 LEADER]: Becoming Leader. State: Replica: a2787a6583a8485d9270dbc40d793edb, State: Running, Role: LEADER
I20260928 17:47:00.337188 9129 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:00.337218 9134 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2787a6583a8485d9270dbc40d793edb" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36137 } }
I20260928 17:47:00.337543 9134 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2787a6583a8485d9270dbc40d793edb. Latest consensus state: current_term: 1 leader_uuid: "a2787a6583a8485d9270dbc40d793edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2787a6583a8485d9270dbc40d793edb" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36137 } } }
I20260928 17:47:00.337548 9135 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2787a6583a8485d9270dbc40d793edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2787a6583a8485d9270dbc40d793edb" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36137 } } }
I20260928 17:47:00.337635 9135 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:00.337774 9134 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:00.337842 9141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:00.338234 9141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:00.339664 9141 catalog_manager.cc:1347] Generated new cluster ID: 7565f9a4d39f449cabe17d1a6c79f410
I20260928 17:47:00.339713 9141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:47:00.359063 9141 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:00.359791 9141 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:00.367053 9141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2787a6583a8485d9270dbc40d793edb: Generated new TSK 0
I20260928 17:47:00.367280 9141 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 17:47:00.391371 9159 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 17:47:00.391462 9159 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 17:47:00.392001 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.375192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.3.222.62:36137)
0928 17:47:00.375257 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.375259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.375593 (+ 334us) server_negotiation.cc:407] Connection header received
0928 17:47:00.375664 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.375664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.375676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.375754 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0928 17:47:00.375990 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.376785 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.377710 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.378026 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.378105 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.378329 (+ 224us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 17:47:00.391707 (+ 13378us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 17:47:00.391912 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 17:47:00.392062 9132 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 17:47:00.392262 9158 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:00.375099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36137 (local address 127.0.0.1:36800)
0928 17:47:00.375513 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.375525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:00.375593 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:00.375845 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:00.375847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:00.375849 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 17:47:00.375909 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:00.375911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.376905 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.376906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:00.377599 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:00.377602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.378106 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.378107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:00.378154 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.378174 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 17:47:00.391855 (+ 13681us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 17:47:00.391956 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.222.62:36137: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":87,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.222.62:36137: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 17:47:00.392733 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 9062
2026-09-28T17:47:00Z chronyd exiting
I20260928 17:47:00.428186 3960 test_util.cc:175] -----------------------------------------------
I20260928 17:47:00.428274 3960 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617599653609-3960-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (252 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 17:47:00.429047 3960 mini_oidc.cc:152] Starting JWKS server
I20260928 17:47:00.429706 3960 webserver.cc:460] Webserver started at https://127.0.0.1:41185/ using document root <none> and password file <none>
I20260928 17:47:00.429754 3960 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 17:47:00.429880 3960 webserver.cc:460] Webserver started at http://127.0.0.1:42107/ using document root <none> and password file <none>
2026-09-28T17:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:00Z Disabled control of system clock
I20260928 17:47:00.437186 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:33719
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:33719
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41185/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/testchainca.cert with env {}
W20260928 17:47:00.515465 9165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:00.515662 9165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:00.515684 9165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:00.516773 9165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:00.516806 9165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:00.516817 9165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:00.516829 9165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:00.516839 9165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 17:47:00.516849 9165 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 17:47:00.516860 9165 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41185/jwks.json
I20260928 17:47:00.518276 9165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:33719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:33719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41185/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:00.518476 9165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:00.518640 9165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:47:00.523182 9165 server_base.cc:1034] running on GCE node
W20260928 17:47:00.523145 9173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:00.523156 9170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:00.523453 9171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:00.523676 9165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:00.523947 9165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:00.525089 9165 hybrid_clock.cc:648] HybridClock initialized: now 1790617620525077 us; error 28 us; skew 500 ppm
I20260928 17:47:00.526243 9165 webserver.cc:460] Webserver started at http://127.3.222.62:46213/ using document root <none> and password file <none>
I20260928 17:47:00.526438 9165 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:00.526486 9165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:00.526605 9165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:00.527654 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "69d1420b80a5458ca0993de1f4b08e0c"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "acebe41c070fe5b52872c9925a4b63dc"
server_key_iv: "9fd72add5eb670e0fa207610194c2d6f"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.527981 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "69d1420b80a5458ca0993de1f4b08e0c"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "acebe41c070fe5b52872c9925a4b63dc"
server_key_iv: "9fd72add5eb670e0fa207610194c2d6f"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.529161 9165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:00.529779 9179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:00.529932 9165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:00.530035 9165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "69d1420b80a5458ca0993de1f4b08e0c"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "acebe41c070fe5b52872c9925a4b63dc"
server_key_iv: "9fd72add5eb670e0fa207610194c2d6f"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.530095 9165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:00.548188 9165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:00.548480 9165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:00.548616 9165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:00.552129 9165 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:33719
I20260928 17:47:00.552163 9231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:33719 every 8 connection(s)
I20260928 17:47:00.552457 9165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:00.553385 9232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:00.555574 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c: Bootstrap starting.
I20260928 17:47:00.556103 9232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:00.556350 9232 log.cc:824] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:00.557036 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c: No bootstrap required, opened a new log
I20260928 17:47:00.558548 9232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d1420b80a5458ca0993de1f4b08e0c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33719 } }
I20260928 17:47:00.558691 9232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:00.558713 9232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d1420b80a5458ca0993de1f4b08e0c, State: Initialized, Role: FOLLOWER
I20260928 17:47:00.558840 9232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d1420b80a5458ca0993de1f4b08e0c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33719 } }
I20260928 17:47:00.558907 9232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:00.558929 9232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:00.558975 9232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:00.559533 9232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d1420b80a5458ca0993de1f4b08e0c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33719 } }
I20260928 17:47:00.559619 9232 leader_election.cc:302] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69d1420b80a5458ca0993de1f4b08e0c; no voters:
I20260928 17:47:00.559762 9232 leader_election.cc:288] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:00.559794 9235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:00.559921 9235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [term 1 LEADER]: Becoming Leader. State: Replica: 69d1420b80a5458ca0993de1f4b08e0c, State: Running, Role: LEADER
I20260928 17:47:00.560002 9232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:00.560016 9235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d1420b80a5458ca0993de1f4b08e0c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33719 } }
I20260928 17:47:00.560402 9236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69d1420b80a5458ca0993de1f4b08e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d1420b80a5458ca0993de1f4b08e0c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33719 } } }
I20260928 17:47:00.560489 9236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:00.560493 9237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d1420b80a5458ca0993de1f4b08e0c. Latest consensus state: current_term: 1 leader_uuid: "69d1420b80a5458ca0993de1f4b08e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d1420b80a5458ca0993de1f4b08e0c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 33719 } } }
I20260928 17:47:00.560586 9237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:00.560714 9244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:00.561192 9244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:00.561245 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 9165
I20260928 17:47:00.561336 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:00.561506 3960 external_mini_cluster.cc:1428] Setting key 86c1ce362d25cf9f0258e3b8706149f6
I20260928 17:47:00.562618 9244 catalog_manager.cc:1347] Generated new cluster ID: 01f1664075e241bebfb497b19c733498
I20260928 17:47:00.562672 9244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:47:00.571192 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.562013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.3.222.62:33719)
0928 17:47:00.562448 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.562456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.562487 (+ 31us) server_negotiation.cc:407] Connection header received
0928 17:47:00.567320 (+ 4833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.567328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.567392 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.567544 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.567742 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.568639 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.569391 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.569809 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.569923 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.570536 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.570540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.570540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.570699 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.570732 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.570742 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.570795 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.570868 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.571022 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":55,"threads_started":1}
I20260928 17:47:00.572100 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:47:00.577323 9244 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:00.577796 9244 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:00.581321 9244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69d1420b80a5458ca0993de1f4b08e0c: Generated new TSK 0
I20260928 17:47:00.581496 9244 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 17:47:00.609756 9262 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 17:47:00.609856 9262 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 17:47:00.610126 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.588616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.3.222.62:33719)
0928 17:47:00.588658 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.588660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.588683 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:47:00.588785 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.588786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.588796 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.588858 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0928 17:47:00.588980 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.589473 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.590196 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.590415 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.590468 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.590627 (+ 159us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 17:47:00.609886 (+ 19259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 17:47:00.610052 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 17:47:00.610195 9254 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 17:47:00.610357 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:00.588525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:33719 (local address 127.0.0.1:60750)
0928 17:47:00.588634 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.588644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:00.588700 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:00.588862 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:00.588863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:00.588865 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 17:47:00.588923 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:00.588926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.589542 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.589543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:00.590121 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:00.590122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.590480 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.590481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:00.590540 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.590546 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 17:47:00.610141 (+ 19595us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 17:47:00.610268 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.222.62:33719: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":68,"threads_started":1}
I20260928 17:47:00.610646 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 9165
2026-09-28T17:47:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 17:47:00.647023 3960 mini_oidc.cc:152] Starting JWKS server
I20260928 17:47:00.647747 3960 webserver.cc:460] Webserver started at https://127.0.0.1:46725/ using document root <none> and password file <none>
I20260928 17:47:00.647791 3960 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 17:47:00.647897 3960 webserver.cc:460] Webserver started at http://127.0.0.1:35927/ using document root <none> and password file <none>
2026-09-28T17:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:00Z Disabled control of system clock
I20260928 17:47:00.654837 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:39971
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:39971
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46725/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 17:47:00.731756 9268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:00.731962 9268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:00.731982 9268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:00.733058 9268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:00.733090 9268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:00.733101 9268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:00.733112 9268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:00.733122 9268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 17:47:00.733134 9268 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 17:47:00.733143 9268 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46725/jwks.json
I20260928 17:47:00.734408 9268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:39971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:39971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46725/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:00.734591 9268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:00.734822 9268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:00.739283 9274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:00.739274 9273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:00.739378 9268 server_base.cc:1034] running on GCE node
W20260928 17:47:00.739272 9276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:00.739737 9268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:00.740005 9268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:00.741169 9268 hybrid_clock.cc:648] HybridClock initialized: now 1790617620741153 us; error 29 us; skew 500 ppm
I20260928 17:47:00.742393 9268 webserver.cc:460] Webserver started at http://127.3.222.62:39715/ using document root <none> and password file <none>
I20260928 17:47:00.742606 9268 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:00.742650 9268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:00.742761 9268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:00.743818 9268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "f67cefbb0d514929b32d38a2ad2d2ef9"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "2e2ae0ad52c108ef59e09154ec5a39e6"
server_key_iv: "30c3897077e02194604da291d34aff74"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.744140 9268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "f67cefbb0d514929b32d38a2ad2d2ef9"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "2e2ae0ad52c108ef59e09154ec5a39e6"
server_key_iv: "30c3897077e02194604da291d34aff74"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.745285 9268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:00.745975 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:00.746133 9268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:00.746203 9268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "f67cefbb0d514929b32d38a2ad2d2ef9"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "2e2ae0ad52c108ef59e09154ec5a39e6"
server_key_iv: "30c3897077e02194604da291d34aff74"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.746268 9268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:00.768424 9268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:00.768694 9268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:00.768842 9268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:00.772838 9268 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:39971
I20260928 17:47:00.772866 9334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:39971 every 8 connection(s)
I20260928 17:47:00.773188 9268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:00.773871 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:00.776299 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9: Bootstrap starting.
I20260928 17:47:00.776804 9335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:00.777026 9335 log.cc:824] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:00.777626 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9: No bootstrap required, opened a new log
I20260928 17:47:00.778503 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 9268
I20260928 17:47:00.778595 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:00.778764 3960 external_mini_cluster.cc:1428] Setting key 0400ca8778eb22c573cabb7ec67013cc
I20260928 17:47:00.779493 9335 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67cefbb0d514929b32d38a2ad2d2ef9" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39971 } }
I20260928 17:47:00.779626 9335 raft_consensus.cc:374] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:00.779657 9335 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67cefbb0d514929b32d38a2ad2d2ef9, State: Initialized, Role: FOLLOWER
I20260928 17:47:00.779775 9335 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67cefbb0d514929b32d38a2ad2d2ef9" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39971 } }
I20260928 17:47:00.779855 9335 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:00.779886 9335 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:00.779929 9335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:00.780550 9335 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67cefbb0d514929b32d38a2ad2d2ef9" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39971 } }
I20260928 17:47:00.780647 9335 leader_election.cc:302] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67cefbb0d514929b32d38a2ad2d2ef9; no voters:
I20260928 17:47:00.780781 9335 leader_election.cc:288] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:00.780825 9340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:00.781005 9340 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [term 1 LEADER]: Becoming Leader. State: Replica: f67cefbb0d514929b32d38a2ad2d2ef9, State: Running, Role: LEADER
I20260928 17:47:00.781023 9335 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:00.781097 9340 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67cefbb0d514929b32d38a2ad2d2ef9" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39971 } }
I20260928 17:47:00.781409 9340 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67cefbb0d514929b32d38a2ad2d2ef9. Latest consensus state: current_term: 1 leader_uuid: "f67cefbb0d514929b32d38a2ad2d2ef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67cefbb0d514929b32d38a2ad2d2ef9" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39971 } } }
I20260928 17:47:00.781600 9340 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:00.781543 9341 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f67cefbb0d514929b32d38a2ad2d2ef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67cefbb0d514929b32d38a2ad2d2ef9" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 39971 } } }
I20260928 17:47:00.781942 9341 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:00.782100 9349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:00.782603 9349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:00.784667 9349 catalog_manager.cc:1347] Generated new cluster ID: 7c8a6601f6b349e18ccdb5badcad5b80
I20260928 17:47:00.784751 9349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:47:00.784798 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.779285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.3.222.62:39971)
0928 17:47:00.779484 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.779492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.779515 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:47:00.779605 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.779612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.779668 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.779799 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.780004 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.781751 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.782787 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.783390 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.783497 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.784141 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.784145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.784145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.784302 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.784334 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.784344 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.784398 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.784491 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.784565 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260928 17:47:00.785635 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:47:00.792596 9349 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:00.793084 9349 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:00.795269 9349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f67cefbb0d514929b32d38a2ad2d2ef9: Generated new TSK 0
I20260928 17:47:00.795408 9349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:47:00.805836 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.802769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.3.222.62:39971)
0928 17:47:00.802817 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.802819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.802904 (+ 85us) server_negotiation.cc:407] Connection header received
0928 17:47:00.802999 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.803000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.803012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.803070 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0928 17:47:00.803186 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.803688 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.804445 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.804631 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.804682 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.804832 (+ 150us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 17:47:00.805651 (+ 819us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 17:47:00.805774 (+ 123us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 17:47:00.805884 9339 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 17:47:00.805959 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:00.802709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:39971 (local address 127.0.0.1:34818)
0928 17:47:00.802842 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:00.802852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:00.802911 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:00.803067 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:00.803068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:00.803070 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 17:47:00.803125 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:00.803128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.803776 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.803777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:00.804364 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:00.804366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.804691 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.804692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:00.804748 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.804754 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 17:47:00.805761 (+ 1007us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 17:47:00.805862 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.222.62:39971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260928 17:47:00.806094 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 9268
E20260928 17:47:00.816443 9365 webserver.cc:533] Webserver: sslize failed:
E20260928 17:47:00.816511 9365 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T17:47:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (195 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T17:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:00Z Disabled control of system clock
I20260928 17:47:00.849505 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:43823
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:43823
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/testchainca.cert with env {}
W20260928 17:47:00.925930 9369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:00.926128 9369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:00.926149 9369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:00.927307 9369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:00.927345 9369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:00.927356 9369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:00.927366 9369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:00.927378 9369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 17:47:00.927402 9369 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 17:47:00.927417 9369 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 17:47:00.928895 9369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:43823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:43823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:00.929111 9369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:00.929297 9369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:00.933809 9374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:00.933809 9375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:00.933835 9377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:00.934074 9369 server_base.cc:1034] running on GCE node
I20260928 17:47:00.934329 9369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:00.934584 9369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:00.935720 9369 hybrid_clock.cc:648] HybridClock initialized: now 1790617620935707 us; error 23 us; skew 500 ppm
I20260928 17:47:00.936775 9369 webserver.cc:460] Webserver started at http://127.3.222.62:46331/ using document root <none> and password file <none>
I20260928 17:47:00.936986 9369 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:00.937032 9369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:00.937150 9369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:00.938113 9369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "5d845178a1764b16ac5691e878aaeb72"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "c1bca93394440c9300df9c7c4e678ab9"
server_key_iv: "08d29efb5ba2d2d8c691027b21424393"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.938421 9369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "5d845178a1764b16ac5691e878aaeb72"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "c1bca93394440c9300df9c7c4e678ab9"
server_key_iv: "08d29efb5ba2d2d8c691027b21424393"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.939600 9369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:47:00.940312 9383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:00.940465 9369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:00.940541 9369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "5d845178a1764b16ac5691e878aaeb72"
format_stamp: "Formatted at 2026-09-28 17:47:00 on dist-test-slave-w3bm"
server_key: "c1bca93394440c9300df9c7c4e678ab9"
server_key_iv: "08d29efb5ba2d2d8c691027b21424393"
server_key_version: "encryptionkey@0"
I20260928 17:47:00.940598 9369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:00.958545 9369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:00.958822 9369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:00.958945 9369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:00.962527 9369 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:43823
I20260928 17:47:00.962570 9435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:43823 every 8 connection(s)
I20260928 17:47:00.962867 9369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:00.963173 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 9369
I20260928 17:47:00.963291 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:00.963459 3960 external_mini_cluster.cc:1428] Setting key eb968319be6e26b92af5b656644da093
I20260928 17:47:00.963891 9436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:00.966962 9436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72: Bootstrap starting.
I20260928 17:47:00.967630 9436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:00.967936 9436 log.cc:824] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:00.968708 9436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72: No bootstrap required, opened a new log
I20260928 17:47:00.969340 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:00.963919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.3.222.62:43823)
0928 17:47:00.964122 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:00.964132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:00.964161 (+ 29us) server_negotiation.cc:407] Connection header received
0928 17:47:00.964641 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:00.964650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:00.964734 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:00.964900 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:00.965032 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.966321 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.967083 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:00.967689 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:00.967813 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:00.968700 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:00.968704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:00.968705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:00.968931 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:00.968966 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:00.968977 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:00.969029 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:00.969121 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 17:47:00.969180 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260928 17:47:00.970183 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:47:00.970746 9436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d845178a1764b16ac5691e878aaeb72" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43823 } }
I20260928 17:47:00.970901 9436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:00.970953 9436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d845178a1764b16ac5691e878aaeb72, State: Initialized, Role: FOLLOWER
I20260928 17:47:00.971064 9436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d845178a1764b16ac5691e878aaeb72" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43823 } }
I20260928 17:47:00.971135 9436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:00.971155 9436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:00.971199 9436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:00.971721 9436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d845178a1764b16ac5691e878aaeb72" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43823 } }
I20260928 17:47:00.971807 9436 leader_election.cc:302] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d845178a1764b16ac5691e878aaeb72; no voters:
I20260928 17:47:00.971954 9436 leader_election.cc:288] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:00.971993 9441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:00.972121 9441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [term 1 LEADER]: Becoming Leader. State: Replica: 5d845178a1764b16ac5691e878aaeb72, State: Running, Role: LEADER
I20260928 17:47:00.972177 9436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:00.972203 9441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d845178a1764b16ac5691e878aaeb72" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43823 } }
I20260928 17:47:00.972553 9442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d845178a1764b16ac5691e878aaeb72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d845178a1764b16ac5691e878aaeb72" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43823 } } }
I20260928 17:47:00.972560 9443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d845178a1764b16ac5691e878aaeb72. Latest consensus state: current_term: 1 leader_uuid: "5d845178a1764b16ac5691e878aaeb72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d845178a1764b16ac5691e878aaeb72" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43823 } } }
I20260928 17:47:00.972703 9442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:00.972712 9443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:00.973078 9450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:00.973526 9450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:00.974906 9450 catalog_manager.cc:1347] Generated new cluster ID: 343d1d3692ce415a8cd5ffcac24de3e8
I20260928 17:47:00.974946 9450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:47:00.991611 9450 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:00.992136 9450 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:00.995536 9450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d845178a1764b16ac5691e878aaeb72: Generated new TSK 0
I20260928 17:47:00.995682 9450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:47:01.090600 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:01.005798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.3.222.62:43823)
0928 17:47:01.005842 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:01.005844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:01.005912 (+ 68us) server_negotiation.cc:407] Connection header received
0928 17:47:01.005964 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:01.005965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:01.005977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:01.006060 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0928 17:47:01.006153 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:01.006925 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:01.007705 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:01.008030 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:01.008096 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:01.008247 (+ 151us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 17:47:01.090327 (+ 82080us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 17:47:01.090523 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59638: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 17:47:01.090677 9439 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59638: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 17:47:01.090674 9465 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:01.005733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:43823 (local address 127.0.0.1:59638)
0928 17:47:01.005845 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:01.005854 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:01.005910 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:01.006056 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:01.006056 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:01.006058 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 17:47:01.006102 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:01.006104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:01.007004 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:01.007005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:01.007631 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:01.007633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:01.008103 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:01.008104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:01.008155 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:01.008161 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 17:47:01.090462 (+ 82301us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 17:47:01.090593 (+ 131us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.222.62:43823: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":71,"threads_started":1}
I20260928 17:47:01.091048 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 9369
2026-09-28T17:47:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (264 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (83 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:47:01 dist-test-slave-w3bm krb5kdc[9477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:47:01 dist-test-slave-w3bm krb5kdc[9477](info): set up 2 sockets
Sep 28 17:47:01 dist-test-slave-w3bm krb5kdc[9477](info): commencing operation
krb5kdc: starting...
W20260928 17:47:02.290995 3960 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 17:47:02.518414 3960 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.227s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 17:47:02.708575 3960 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.190s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 17:47:02.961717 3960 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.253s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/test-user.keytab.
W20260928 17:47:03.929921 3960 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.968s user 0.000s sys 0.001s
Sep 28 17:47:03 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
W20260928 17:47:04.283946 3960 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.222.62: real 0.334s user 0.000s sys 0.002s
I20260928 17:47:04.284315 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38399
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:38399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:04.365799 9490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:04.365989 9490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:04.366008 9490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:04.367077 9490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:04.367108 9490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:04.367121 9490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:04.367132 9490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:04.367142 9490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:04.368563 9490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:38399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:04.368743 9490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:04.368903 9490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:04.373369 9496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:04.373405 9495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:04.373497 9498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:04.373610 9490 server_base.cc:1034] running on GCE node
I20260928 17:47:04.373808 9490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:04.374061 9490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:04.375198 9490 hybrid_clock.cc:648] HybridClock initialized: now 1790617624375191 us; error 29 us; skew 500 ppm
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.377477 9490 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:47:04.377876 9490 webserver.cc:460] Webserver started at http://127.3.222.62:39409/ using document root <none> and password file <none>
I20260928 17:47:04.378087 9490 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:04.378132 9490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:04.378248 9490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:04.379272 9490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "fcc5526a40884703bfbcc8e7c98c0fcd"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "e832beb856118becbe1da52f0941534c"
server_key_iv: "49fd125426536c02c97d360e732c108f"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.379608 9490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "fcc5526a40884703bfbcc8e7c98c0fcd"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "e832beb856118becbe1da52f0941534c"
server_key_iv: "49fd125426536c02c97d360e732c108f"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.380774 9490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:47:04.381488 9505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.381628 9490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:04.381708 9490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "fcc5526a40884703bfbcc8e7c98c0fcd"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "e832beb856118becbe1da52f0941534c"
server_key_iv: "49fd125426536c02c97d360e732c108f"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.381771 9490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:04.398307 9490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:04.400234 9490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:04.400370 9490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:04.404057 9490 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:38399
I20260928 17:47:04.404100 9557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:38399 every 8 connection(s)
I20260928 17:47:04.404423 9490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:04.405210 9558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:04.407394 9558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd: Bootstrap starting.
I20260928 17:47:04.407922 9558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:04.408166 9558 log.cc:824] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:04.408737 9558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd: No bootstrap required, opened a new log
I20260928 17:47:04.408795 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 9490
I20260928 17:47:04.408892 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:04.409052 3960 external_mini_cluster.cc:1428] Setting key c21894927c3ba1c694378f05236b7966
I20260928 17:47:04.410691 9558 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc5526a40884703bfbcc8e7c98c0fcd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38399 } }
I20260928 17:47:04.410866 9558 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:04.410939 9558 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc5526a40884703bfbcc8e7c98c0fcd, State: Initialized, Role: FOLLOWER
I20260928 17:47:04.411067 9558 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc5526a40884703bfbcc8e7c98c0fcd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38399 } }
I20260928 17:47:04.411137 9558 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:04.411160 9558 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:04.411203 9558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:04.412211 9558 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc5526a40884703bfbcc8e7c98c0fcd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38399 } }
I20260928 17:47:04.412314 9558 leader_election.cc:302] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcc5526a40884703bfbcc8e7c98c0fcd; no voters:
I20260928 17:47:04.412451 9558 leader_election.cc:288] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:04.412480 9563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:04.412609 9563 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [term 1 LEADER]: Becoming Leader. State: Replica: fcc5526a40884703bfbcc8e7c98c0fcd, State: Running, Role: LEADER
I20260928 17:47:04.412667 9558 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:04.412729 9563 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc5526a40884703bfbcc8e7c98c0fcd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38399 } }
I20260928 17:47:04.413050 9564 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcc5526a40884703bfbcc8e7c98c0fcd. Latest consensus state: current_term: 1 leader_uuid: "fcc5526a40884703bfbcc8e7c98c0fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc5526a40884703bfbcc8e7c98c0fcd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38399 } } }
I20260928 17:47:04.413183 9564 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:04.413715 9572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:04.414089 9565 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcc5526a40884703bfbcc8e7c98c0fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc5526a40884703bfbcc8e7c98c0fcd" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38399 } } }
I20260928 17:47:04.414160 9572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:04.414224 9565 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:04.415612 9572 catalog_manager.cc:1347] Generated new cluster ID: 66c740a2b4a740d5af02cf7f2176be7a
I20260928 17:47:04.415652 9572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:04.420657 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.409529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.3.222.62:38399)
0928 17:47:04.409709 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.409718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.409750 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:47:04.410662 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.410670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.410734 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.410879 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.411567 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.412906 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.413916 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.414479 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.414589 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.417281 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:04.417286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:04.417286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:04.417298 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:04.419093 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.419476 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.419479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.419480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.419597 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.419790 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.419791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.419792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.420098 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:04.420248 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:04.420326 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 17:47:04.420463 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260928 17:47:04.422031 9572 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:04.422525 9572 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:04.425190 9572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcc5526a40884703bfbcc8e7c98c0fcd: Generated new TSK 0
I20260928 17:47:04.425341 9572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:47:04.454495 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38399
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:04.533066 9584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:04.533257 9584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:04.533288 9584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:04.534394 9584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:04.534432 9584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:04.534466 9584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:47:04.535923 9584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:04.536141 9584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:04.536338 9584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:04.540935 9589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:04.540943 9592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:04.540943 9590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:04.540970 9584 server_base.cc:1034] running on GCE node
I20260928 17:47:04.541321 9584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:04.541497 9584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:04.542625 9584 hybrid_clock.cc:648] HybridClock initialized: now 1790617624542612 us; error 29 us; skew 500 ppm
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.544888 9584 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:47:04.545241 9584 webserver.cc:460] Webserver started at http://127.3.222.1:36743/ using document root <none> and password file <none>
I20260928 17:47:04.545408 9584 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:04.545442 9584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:04.545521 9584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:04.546435 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "326b4782f1c945bf82941d3f2acff7a0"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "ae39d327f00282a19c5a074e1d88d264"
server_key_iv: "75caee04d3f1f7c12a05e6f1de60c8e2"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.546706 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "326b4782f1c945bf82941d3f2acff7a0"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "ae39d327f00282a19c5a074e1d88d264"
server_key_iv: "75caee04d3f1f7c12a05e6f1de60c8e2"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.547852 9584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:04.548529 9599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.548676 9584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:04.548758 9584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "326b4782f1c945bf82941d3f2acff7a0"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "ae39d327f00282a19c5a074e1d88d264"
server_key_iv: "75caee04d3f1f7c12a05e6f1de60c8e2"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.548836 9584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:04.561672 9584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:04.563592 9584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:04.563719 9584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:04.563925 9584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:04.564185 9584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:04.564220 9584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.564246 9584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:04.564268 9584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.569514 9584 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:43273
I20260928 17:47:04.569564 9712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:43273 every 8 connection(s)
I20260928 17:47:04.569833 9584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:04.578078 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.570487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:35587 (local address 127.3.222.62:38399)
0928 17:47:04.570538 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.570540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.570716 (+ 176us) server_negotiation.cc:407] Connection header received
0928 17:47:04.571674 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.571676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.571691 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.571738 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.572314 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.572821 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.573755 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.573967 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.574019 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.575931 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:04.575933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:04.575933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:04.575941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:04.577113 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.577420 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.577421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.577421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.577463 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.577625 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.577625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.577625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.577715 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:04.577757 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:04.577953 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 17:47:04.578014 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:47:04.578274 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.570470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38399 (local address 127.3.222.1:35587)
0928 17:47:04.570611 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:04.570646 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:04.571582 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:04.571816 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:04.571820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:04.572053 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:04.572229 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:04.572232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.572896 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.572897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:04.573679 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:04.573682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.574031 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.574032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:04.574147 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.574554 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:04.574562 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:04.575843 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:04.577173 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:04.577178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:04.577179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:04.577367 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:04.577504 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:04.577505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:04.577505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:04.577586 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:04.577754 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:04.577756 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:04.577863 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 17:47:04.577934 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 17:47:04.578052 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260928 17:47:04.578699 9713 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38399
I20260928 17:47:04.578776 9713 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:04.578909 9713 heartbeater.cc:511] Master 127.3.222.62:38399 requested a full tablet report, sending...
I20260928 17:47:04.579021 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 9584
I20260928 17:47:04.579115 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:47:04.579340 3960 external_mini_cluster.cc:1428] Setting key 8413f90dda28a88bb6702d6437a2f84e
I20260928 17:47:04.579416 9522 ts_manager.cc:194] Registered new tserver with Master: 326b4782f1c945bf82941d3f2acff7a0 (127.3.222.1:43273)
I20260928 17:47:04.580310 9522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:35587
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:47:04.601764 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38399
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:04.678743 9718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:04.678915 9718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:04.678936 9718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:04.680052 9718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:04.680091 9718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:04.680116 9718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:47:04.681520 9718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:04.681689 9718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:04.681855 9718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:04.686535 9726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:04.686529 9724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:04.686529 9723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:04.686748 9718 server_base.cc:1034] running on GCE node
I20260928 17:47:04.686971 9718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:04.687150 9718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:04.688292 9718 hybrid_clock.cc:648] HybridClock initialized: now 1790617624688282 us; error 27 us; skew 500 ppm
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.690501 9718 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:47:04.690899 9718 webserver.cc:460] Webserver started at http://127.3.222.2:39423/ using document root <none> and password file <none>
I20260928 17:47:04.691093 9718 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:04.691138 9718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:04.691290 9718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:04.692217 9718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "8c77351a5883412b851c1058642a32ef"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "61d88b9f387235070ff427c0c15b9c54"
server_key_iv: "2d460f7dd838a34c530c61482a9d45d1"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.692512 9718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "8c77351a5883412b851c1058642a32ef"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "61d88b9f387235070ff427c0c15b9c54"
server_key_iv: "2d460f7dd838a34c530c61482a9d45d1"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.693645 9718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:47:04.694319 9733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.694475 9718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:04.694551 9718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "8c77351a5883412b851c1058642a32ef"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "61d88b9f387235070ff427c0c15b9c54"
server_key_iv: "2d460f7dd838a34c530c61482a9d45d1"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.694612 9718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:04.708731 9718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:04.710420 9718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:04.710537 9718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:04.710719 9718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:04.710968 9718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:04.710999 9718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.711022 9718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:04.711050 9718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.716403 9718 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:42673
I20260928 17:47:04.716456 9846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:42673 every 8 connection(s)
I20260928 17:47:04.716710 9718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:04.724918 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.717275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:43121 (local address 127.3.222.62:38399)
0928 17:47:04.717326 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.717328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.717485 (+ 157us) server_negotiation.cc:407] Connection header received
0928 17:47:04.718378 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.718380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.718393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.718437 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.718989 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.719551 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.720513 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.720721 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.720776 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.722646 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:04.722648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:04.722648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:04.722656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:04.723787 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.724112 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.724114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.724114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.724163 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.724360 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.724361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.724362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.724462 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:04.724515 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:04.724750 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 17:47:04.724831 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:47:04.725047 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.717257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38399 (local address 127.3.222.2:43121)
0928 17:47:04.717380 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:04.717411 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:04.718275 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:04.718511 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:04.718515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:04.718737 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:04.718905 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:04.718909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.719625 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.719626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:04.720432 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:04.720434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.720786 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.720787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:04.720896 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.721315 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:04.721322 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:04.722560 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:04.723854 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:04.723860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:04.723862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:04.724051 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:04.724219 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:04.724220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:04.724220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:04.724309 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:04.724513 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:04.724518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:04.724637 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 17:47:04.724724 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 17:47:04.724859 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260928 17:47:04.725504 9847 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38399
I20260928 17:47:04.725584 9847 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:04.725732 9847 heartbeater.cc:511] Master 127.3.222.62:38399 requested a full tablet report, sending...
I20260928 17:47:04.725906 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 9718
I20260928 17:47:04.725999 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:47:04.726104 9522 ts_manager.cc:194] Registered new tserver with Master: 8c77351a5883412b851c1058642a32ef (127.3.222.2:42673)
I20260928 17:47:04.726176 3960 external_mini_cluster.cc:1428] Setting key 4bf2a1b512581f2d25de0deaeb71b67e
I20260928 17:47:04.726684 9522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:43121
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:47:04.749415 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38399
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:04.826488 9852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:04.826678 9852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:04.826699 9852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:04.827790 9852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:04.827828 9852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:04.827852 9852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:04.829217 9852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:04.829396 9852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:04.829558 9852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:04.834228 9860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:04.834261 9858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:04.834239 9857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:04.834259 9852 server_base.cc:1034] running on GCE node
I20260928 17:47:04.834718 9852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:04.834928 9852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:04.836090 9852 hybrid_clock.cc:648] HybridClock initialized: now 1790617624836079 us; error 30 us; skew 500 ppm
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.838577 9852 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:47:04.838995 9852 webserver.cc:460] Webserver started at http://127.3.222.3:33263/ using document root <none> and password file <none>
I20260928 17:47:04.839190 9852 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:04.839265 9852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:04.839391 9852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:04.840358 9852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "7f22260f9d0145838008baeea65b9882"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "3df3a1a3afe0e5733acc1dbced0c1963"
server_key_iv: "1880bea8243368823287a671c35e5d4b"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.840665 9852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "7f22260f9d0145838008baeea65b9882"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "3df3a1a3afe0e5733acc1dbced0c1963"
server_key_iv: "1880bea8243368823287a671c35e5d4b"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.841817 9852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:04.842432 9867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.842573 9852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:04.842648 9852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "7f22260f9d0145838008baeea65b9882"
format_stamp: "Formatted at 2026-09-28 17:47:04 on dist-test-slave-w3bm"
server_key: "3df3a1a3afe0e5733acc1dbced0c1963"
server_key_iv: "1880bea8243368823287a671c35e5d4b"
server_key_version: "encryptionkey@0"
I20260928 17:47:04.842710 9852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:04.862879 9852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:04.864651 9852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:04.864778 9852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:04.864987 9852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:04.865262 9852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:04.865322 9852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.865360 9852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:04.865386 9852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:04.870572 9852 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:32991
I20260928 17:47:04.870626 9980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:32991 every 8 connection(s)
I20260928 17:47:04.870896 9852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:47:04.873214 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 9852
I20260928 17:47:04.873309 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:47:04.873497 3960 external_mini_cluster.cc:1428] Setting key 17d98b8985cacf5910e63796c7263349
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:04.879434 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.871544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:53831 (local address 127.3.222.62:38399)
0928 17:47:04.871587 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.871589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.871778 (+ 189us) server_negotiation.cc:407] Connection header received
0928 17:47:04.872679 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.872680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.872693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.872743 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.873342 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.874031 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.875035 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.875250 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.875318 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.877290 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:04.877291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:04.877292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:04.877304 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:04.878386 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.878726 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.878727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.878727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.878778 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.878944 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.878944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.878945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.879034 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:04.879077 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:04.879312 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 17:47:04.879370 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:47:04.879560 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.871525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38399 (local address 127.3.222.3:53831)
0928 17:47:04.871668 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:04.871702 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:04.872581 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:04.872819 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:04.872823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:04.873067 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:04.873249 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:04.873253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.874125 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.874126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:04.874973 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:04.874976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.875329 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.875330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:04.875432 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.875880 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:04.875887 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:04.877206 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:04.878453 (+ 1247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:04.878458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:04.878460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:04.878659 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:04.878820 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:04.878820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:04.878821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:04.878903 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:04.879074 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:04.879077 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:04.879179 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 17:47:04.879270 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 17:47:04.879387 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260928 17:47:04.879981 9981 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38399
I20260928 17:47:04.880061 9981 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:04.880203 9981 heartbeater.cc:511] Master 127.3.222.62:38399 requested a full tablet report, sending...
I20260928 17:47:04.880533 9522 ts_manager.cc:194] Registered new tserver with Master: 7f22260f9d0145838008baeea65b9882 (127.3.222.3:32991)
I20260928 17:47:04.880878 9522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:53831
I20260928 17:47:04.884977 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/test-admin.keytab.
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.900120 3960 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 17:47:04.901741 9988 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:04.900907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.3.222.1:43273)
0928 17:47:04.901154 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.901158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.901168 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:04.901193 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.901195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.901210 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.901301 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.901650 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59416: BlockingRecv error: recv got EOF from 127.0.0.1:59416 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
W20260928 17:47:04.901762 3960 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5cc)
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.902709 3960 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.903535 3960 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:47:04.910281 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.903999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.3.222.1:43273)
0928 17:47:04.904035 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.904037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.904087 (+ 50us) server_negotiation.cc:407] Connection header received
0928 17:47:04.904125 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.904126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.904136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.904178 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.904503 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.905210 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.905877 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.906182 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.906232 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.908094 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:04.908101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:04.908101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:04.908107 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:04.909446 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.909731 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.909733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.909733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.909780 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.909909 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.909910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.909910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.910016 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:04.910102 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:04.910154 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 17:47:04.910211 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:47:04.911768 9988 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:04.911357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.3.222.1:43273)
0928 17:47:04.911395 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.911396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.911433 (+ 37us) server_negotiation.cc:407] Connection header received
0928 17:47:04.911475 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.911476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.911484 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.911533 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.911728 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59436: BlockingRecv error: recv got EOF from 127.0.0.1:59436 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 17:47:04.911900 3960 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5cc' of type FILE
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.912714 3960 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.913430 3960 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:47:04.919560 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.913859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.3.222.1:43273)
0928 17:47:04.913893 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.913894 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.913924 (+ 30us) server_negotiation.cc:407] Connection header received
0928 17:47:04.913972 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.913972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.913981 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.914019 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.914349 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.914805 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.915504 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.915699 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.915750 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.917466 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:04.917468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:04.917468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:04.917475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:04.918469 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.918739 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.918740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.918741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.918871 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.919078 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.919079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.919079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.919173 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:04.919310 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:04.919402 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 17:47:04.919475 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 17:47:04.920849 9988 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:04.920324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.3.222.1:43273)
0928 17:47:04.920372 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.920374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.920416 (+ 42us) server_negotiation.cc:407] Connection header received
0928 17:47:04.920458 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.920459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.920482 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.920520 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.920809 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59442: BlockingRecv error: recv got EOF from 127.0.0.1:59442 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 17:47:04.920943 3960 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617599653609-3960-0/krb5kdc/krb5cc' of type FILE
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.921757 3960 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:04.922470 3960 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[9477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:47:04.928577 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:04.922915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.3.222.1:43273)
0928 17:47:04.922965 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:04.922966 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:04.922976 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:04.923005 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:04.923006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:04.923014 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:04.923052 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:04.923459 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.923911 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.924554 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:04.924730 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:04.924785 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:04.926601 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:04.926602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:04.926603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:04.926609 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:04.927675 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.927951 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.927952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.927953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.927997 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:04.928146 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:04.928147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:04.928147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:04.928226 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:04.928311 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:04.928435 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 17:47:04.928500 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:47:04.929106 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 9584
I20260928 17:47:04.933233 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 9718
I20260928 17:47:04.936861 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 9852
I20260928 17:47:04.940307 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 9490
2026-09-28T17:47:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3682 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[10001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[10001](info): set up 2 sockets
Sep 28 17:47:04 dist-test-slave-w3bm krb5kdc[10001](info): commencing operation
krb5kdc: starting...
W20260928 17:47:05.882994 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:47:05 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.62.keytab.
Entry for principal oryx/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.62.keytab.
I20260928 17:47:05.994776 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41825
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.62.keytab
--principal=oryx/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:06.071004 10014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:06.071185 10014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:06.071235 10014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:06.072310 10014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:06.072348 10014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:06.072366 10014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:06.072382 10014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:06.072398 10014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:06.073801 10014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.62.keytab
--principal=oryx/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:06.074016 10014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:06.074198 10014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:06.078557 10019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:06.078558 10020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.078809 10014 server_base.cc:1034] running on GCE node
W20260928 17:47:06.078594 10022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.079083 10014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:06.079363 10014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:06.080508 10014 hybrid_clock.cc:648] HybridClock initialized: now 1790617626080492 us; error 29 us; skew 500 ppm
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:06.082772 10014 init.cc:377] Logged in from keytab as oryx/127.3.222.62@KRBTEST.COM (short username oryx)
I20260928 17:47:06.083163 10014 webserver.cc:460] Webserver started at http://127.3.222.62:41241/ using document root <none> and password file <none>
I20260928 17:47:06.083384 10014 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:06.083428 10014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:06.083545 10014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:06.084551 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "fdf0798ded714fc493705c56ef52d4cf"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "f1efd40d6f5a34243fc75ae591622f6c"
server_key_iv: "4eb2bfa4d81522e589d51131807cb28c"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.084872 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "fdf0798ded714fc493705c56ef52d4cf"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "f1efd40d6f5a34243fc75ae591622f6c"
server_key_iv: "4eb2bfa4d81522e589d51131807cb28c"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.086056 10014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:06.086726 10029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.086870 10014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.086946 10014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "fdf0798ded714fc493705c56ef52d4cf"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "f1efd40d6f5a34243fc75ae591622f6c"
server_key_iv: "4eb2bfa4d81522e589d51131807cb28c"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.087009 10014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:06.098412 10014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:06.100292 10014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:06.100414 10014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:06.103922 10014 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:41825
I20260928 17:47:06.103951 10081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:41825 every 8 connection(s)
I20260928 17:47:06.104233 10014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:06.105149 10082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:06.107304 10082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: Bootstrap starting.
I20260928 17:47:06.107826 10082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:06.108062 10082 log.cc:824] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:06.108690 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 10014
I20260928 17:47:06.108786 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:06.108843 10082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: No bootstrap required, opened a new log
I20260928 17:47:06.108982 3960 external_mini_cluster.cc:1428] Setting key dbc5fe2745701e0e15ed70cfbb480546
I20260928 17:47:06.111096 10082 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } }
I20260928 17:47:06.111268 10082 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:06.111297 10082 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf0798ded714fc493705c56ef52d4cf, State: Initialized, Role: FOLLOWER
I20260928 17:47:06.111420 10082 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } }
I20260928 17:47:06.111490 10082 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:06.111512 10082 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:06.111554 10082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:06.112074 10082 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } }
I20260928 17:47:06.112156 10082 leader_election.cc:302] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdf0798ded714fc493705c56ef52d4cf; no voters:
I20260928 17:47:06.112298 10082 leader_election.cc:288] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:06.112329 10087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:06.112465 10087 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 1 LEADER]: Becoming Leader. State: Replica: fdf0798ded714fc493705c56ef52d4cf, State: Running, Role: LEADER
I20260928 17:47:06.112537 10082 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:06.112594 10087 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } }
I20260928 17:47:06.112994 10087 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf0798ded714fc493705c56ef52d4cf. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } }
I20260928 17:47:06.113085 10087 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:06.113408 10089 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } }
I20260928 17:47:06.113536 10089 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:06.113878 10094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:06.114370 10094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:06.115625 10094 catalog_manager.cc:1347] Generated new cluster ID: 924fdf473e524903a4fe69f6e2ebc043
I20260928 17:47:06.115672 10094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:06.120072 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.109437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.3.222.62:41825)
0928 17:47:06.109597 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.109605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.109628 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:47:06.110130 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.110136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.110181 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.110294 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.111048 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.112321 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.113311 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.113849 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.113948 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.116561 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.116564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.116565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.116576 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.118418 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.118800 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.118804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.118804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.118911 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.119143 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.119144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.119144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.119488 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.119644 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.119748 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.119881 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260928 17:47:06.131747 10094 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:06.132258 10094 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:06.138825 10094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: Generated new TSK 0
I20260928 17:47:06.138994 10094 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab.
Entry for principal oryx/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab.
I20260928 17:47:06.161607 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41825
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab
--principal=oryx/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:06.237830 10108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:06.238011 10108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:06.238040 10108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:06.239135 10108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:06.239176 10108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:06.239228 10108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:47:06.240617 10108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab
--principal=oryx/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:06.240837 10108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:06.241034 10108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:06.245584 10113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:06.245654 10116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:06.245576 10114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.245810 10108 server_base.cc:1034] running on GCE node
I20260928 17:47:06.245961 10108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:06.246146 10108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:06.247279 10108 hybrid_clock.cc:648] HybridClock initialized: now 1790617626247267 us; error 25 us; skew 500 ppm
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:06.249624 10108 init.cc:377] Logged in from keytab as oryx/127.3.222.1@KRBTEST.COM (short username oryx)
I20260928 17:47:06.250010 10108 webserver.cc:460] Webserver started at http://127.3.222.1:45907/ using document root <none> and password file <none>
I20260928 17:47:06.250202 10108 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:06.250247 10108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:06.250387 10108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:06.251391 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "47845caf59fc4a6f8253c8b6acd59276"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "8d2e43741bf09b87a876ce1b2cb58ad0"
server_key_iv: "35c7d2f8db680d6da33c5383f72c4f29"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.251693 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "47845caf59fc4a6f8253c8b6acd59276"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "8d2e43741bf09b87a876ce1b2cb58ad0"
server_key_iv: "35c7d2f8db680d6da33c5383f72c4f29"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.252892 10108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:06.253595 10123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.254815 10108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:06.254897 10108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "47845caf59fc4a6f8253c8b6acd59276"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "8d2e43741bf09b87a876ce1b2cb58ad0"
server_key_iv: "35c7d2f8db680d6da33c5383f72c4f29"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.254963 10108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:06.292049 10108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:06.293852 10108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:06.293972 10108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:06.294170 10108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:06.294471 10108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:06.294507 10108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.294535 10108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:06.294567 10108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.299844 10108 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:36007
I20260928 17:47:06.299898 10236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:36007 every 8 connection(s)
I20260928 17:47:06.300164 10108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:06.305883 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 10108
I20260928 17:47:06.305967 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:47:06.306144 3960 external_mini_cluster.cc:1428] Setting key a704695e31dab1ad825ce431069fa0fa
I20260928 17:47:06.308943 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.300764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:53937 (local address 127.3.222.62:41825)
0928 17:47:06.300804 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.300806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.300981 (+ 175us) server_negotiation.cc:407] Connection header received
0928 17:47:06.301437 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.301438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.301446 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.301496 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.302415 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.302936 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.303988 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.304191 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.304245 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.306267 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.306269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.306269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.306276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.307724 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.308085 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.308087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.308087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.308146 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.308349 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.308350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.308350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.308458 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.308511 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.308765 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.308847 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:47:06.309087 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.300735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.1:53937)
0928 17:47:06.300878 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:06.300918 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:06.301367 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:06.301551 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:06.301555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:06.302150 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:06.302337 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.302342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.303043 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.303044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.303908 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.303911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.304284 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.304285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.304401 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.304838 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:06.304845 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:06.306174 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:06.307805 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.307812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.307814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.308018 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.308199 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.308200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.308201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.308295 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.308510 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:06.308526 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:06.308651 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 17:47:06.308737 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 17:47:06.308879 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260928 17:47:06.309602 10237 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41825
I20260928 17:47:06.309700 10237 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:06.309877 10237 heartbeater.cc:511] Master 127.3.222.62:41825 requested a full tablet report, sending...
I20260928 17:47:06.310364 10046 ts_manager.cc:194] Registered new tserver with Master: 47845caf59fc4a6f8253c8b6acd59276 (127.3.222.1:36007)
I20260928 17:47:06.310957 10046 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:53937
WARNING: no policy specified for oryx/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab.
Entry for principal oryx/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab.
I20260928 17:47:06.330178 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41825
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab
--principal=oryx/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:06.406821 10242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:06.407013 10242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:06.407034 10242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:06.408151 10242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:06.408191 10242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:06.408215 10242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:47:06.409578 10242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab
--principal=oryx/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:06.409754 10242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:06.409917 10242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:06.414464 10248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:06.414492 10247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.414618 10242 server_base.cc:1034] running on GCE node
W20260928 17:47:06.414489 10250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.414846 10242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:06.415052 10242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:06.416219 10242 hybrid_clock.cc:648] HybridClock initialized: now 1790617626416198 us; error 34 us; skew 500 ppm
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:06.418448 10242 init.cc:377] Logged in from keytab as oryx/127.3.222.2@KRBTEST.COM (short username oryx)
I20260928 17:47:06.418859 10242 webserver.cc:460] Webserver started at http://127.3.222.2:41145/ using document root <none> and password file <none>
I20260928 17:47:06.419044 10242 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:06.419094 10242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:06.419226 10242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:06.420168 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "ca5c6d1a25304fe79e860d08d35d55df"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "7a70a00d3ba4edc76890d45de46b93e3"
server_key_iv: "0eb0350dd18f145d0a501483e53d3958"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.420475 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "ca5c6d1a25304fe79e860d08d35d55df"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "7a70a00d3ba4edc76890d45de46b93e3"
server_key_iv: "0eb0350dd18f145d0a501483e53d3958"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.421634 10242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:06.422233 10257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.422382 10242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:06.422456 10242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "ca5c6d1a25304fe79e860d08d35d55df"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "7a70a00d3ba4edc76890d45de46b93e3"
server_key_iv: "0eb0350dd18f145d0a501483e53d3958"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.422513 10242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:06.450898 10242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:06.452967 10242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:06.453106 10242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:06.453305 10242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:06.453580 10242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:06.453613 10242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.453640 10242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:06.453665 10242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.459014 10242 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:46439
I20260928 17:47:06.459076 10370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:46439 every 8 connection(s)
I20260928 17:47:06.459369 10242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:47:06.464355 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 10242
I20260928 17:47:06.464440 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:47:06.464648 3960 external_mini_cluster.cc:1428] Setting key 505a8a27118ec7ed42bafe77ce41b9c9
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:06.467595 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.460005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:40495 (local address 127.3.222.62:41825)
0928 17:47:06.460063 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.460065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.460259 (+ 194us) server_negotiation.cc:407] Connection header received
0928 17:47:06.460718 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.460718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.460730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.460778 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.461598 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.462089 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.463012 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.463197 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.463262 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.465174 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.465175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.465175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.465182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.466312 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.466669 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.466670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.466671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.466727 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.466951 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.466951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.466952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.467079 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.467136 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.467462 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.467531 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 17:47:06.467790 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.459984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.2:40495)
0928 17:47:06.460136 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:06.460170 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:06.460641 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:06.460830 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:06.460833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:06.461347 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:06.461526 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.461531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.462149 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.462149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.462941 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.462943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.463274 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.463275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.463381 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.463786 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:06.463795 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:06.465113 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:06.466380 (+ 1267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.466386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.466388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.466605 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.466778 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.466779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.466779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.466895 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.467137 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:06.467144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:06.467322 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 17:47:06.467425 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 17:47:06.467586 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260928 17:47:06.468181 10371 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41825
I20260928 17:47:06.468271 10371 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:06.468423 10371 heartbeater.cc:511] Master 127.3.222.62:41825 requested a full tablet report, sending...
I20260928 17:47:06.468783 10046 ts_manager.cc:194] Registered new tserver with Master: ca5c6d1a25304fe79e860d08d35d55df (127.3.222.2:46439)
I20260928 17:47:06.469137 10046 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:40495
WARNING: no policy specified for oryx/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab.
Entry for principal oryx/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab.
I20260928 17:47:06.489418 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41825
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab
--principal=oryx/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:06.565830 10376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:06.566044 10376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:06.566074 10376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:06.567168 10376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:06.567226 10376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:06.567265 10376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:06.568689 10376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab
--principal=oryx/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:06.568917 10376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:06.569108 10376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:06.574252 10381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.574311 10376 server_base.cc:1034] running on GCE node
W20260928 17:47:06.574261 10384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:06.574261 10382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.574678 10376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:06.574863 10376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:06.576014 10376 hybrid_clock.cc:648] HybridClock initialized: now 1790617626575998 us; error 28 us; skew 500 ppm
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:06.578285 10376 init.cc:377] Logged in from keytab as oryx/127.3.222.3@KRBTEST.COM (short username oryx)
I20260928 17:47:06.578707 10376 webserver.cc:460] Webserver started at http://127.3.222.3:36367/ using document root <none> and password file <none>
I20260928 17:47:06.578891 10376 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:06.578938 10376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:06.579052 10376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:06.580086 10376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "f45792f6552b400b817ec4742a71836a"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "209819d69f51c291dce4207ceae9cac7"
server_key_iv: "ba59946f2834f2f0fcb52cf95ca66005"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.580394 10376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "f45792f6552b400b817ec4742a71836a"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "209819d69f51c291dce4207ceae9cac7"
server_key_iv: "ba59946f2834f2f0fcb52cf95ca66005"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.581575 10376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:06.582195 10391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.582337 10376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:06.582414 10376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "f45792f6552b400b817ec4742a71836a"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "209819d69f51c291dce4207ceae9cac7"
server_key_iv: "ba59946f2834f2f0fcb52cf95ca66005"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.582477 10376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:06.608625 10376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:06.610383 10376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:06.610518 10376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:06.610728 10376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:06.611012 10376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:06.611063 10376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.611099 10376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:06.611122 10376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.616755 10376 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:35127
I20260928 17:47:06.616791 10504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:35127 every 8 connection(s)
I20260928 17:47:06.617071 10376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:06.623596 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 10376
I20260928 17:47:06.623684 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:47:06.623845 3960 external_mini_cluster.cc:1428] Setting key 0ab233fcb57be8bbf6ce0a56c0c3e0ed
I20260928 17:47:06.625424 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.617669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:36541 (local address 127.3.222.62:41825)
0928 17:47:06.617720 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.617721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.617894 (+ 173us) server_negotiation.cc:407] Connection header received
0928 17:47:06.618357 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.618359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.618370 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.618420 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.619345 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.619844 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.620785 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.620978 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.621038 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.622845 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.622846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.622847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.622854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.624206 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.624630 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.624631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.624632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.624707 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.624926 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.624926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.624927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.625049 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.625096 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.625308 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.625364 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:47:06.625587 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.617647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.3:36541)
0928 17:47:06.617790 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:06.617823 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:06.618274 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:06.618484 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:06.618487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:06.619043 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:06.619266 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.619271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.619919 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.619920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.620712 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.620714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.621040 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.621041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.621156 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.621570 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:06.621577 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:06.622765 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:06.624284 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.624290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.624292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.624567 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.624761 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.624762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.624762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.624874 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.625096 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:06.625099 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:06.625208 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 17:47:06.625289 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 17:47:06.625405 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260928 17:47:06.625985 10505 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41825
I20260928 17:47:06.626088 10505 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:06.626243 10505 heartbeater.cc:511] Master 127.3.222.62:41825 requested a full tablet report, sending...
I20260928 17:47:06.626585 10046 ts_manager.cc:194] Registered new tserver with Master: f45792f6552b400b817ec4742a71836a (127.3.222.3:35127)
I20260928 17:47:06.626941 10046 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:36541
I20260928 17:47:06.635584 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.222.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.61.keytab.
Entry for principal oryx/127.3.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.61.keytab.
I20260928 17:47:06.658239 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.61:45657
--webserver_interface=127.3.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.61.keytab
--principal=oryx/127.3.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41825,127.3.222.61:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:06.735647 10510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:06.735841 10510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:06.735862 10510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:06.736956 10510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:06.736995 10510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:06.737007 10510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:06.737018 10510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:06.737039 10510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:06.738443 10510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41825,127.3.222.61:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.61.keytab
--principal=oryx/127.3.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.61:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
W20260928 17:47:06.738535 10510 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.222.62:41825,127.3.222.61:45657'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 17:47:06.738670 10510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:06.738859 10510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:06.743325 10515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.743414 10510 server_base.cc:1034] running on GCE node
W20260928 17:47:06.743325 10516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:06.743558 10518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.743813 10510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:06.744062 10510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:06.745191 10510 hybrid_clock.cc:648] HybridClock initialized: now 1790617626745180 us; error 28 us; skew 500 ppm
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:06.747700 10510 init.cc:377] Logged in from keytab as oryx/127.3.222.61@KRBTEST.COM (short username oryx)
I20260928 17:47:06.748071 10510 webserver.cc:460] Webserver started at http://127.3.222.61:41987/ using document root <none> and password file <none>
I20260928 17:47:06.748255 10510 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:06.748298 10510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:06.748428 10510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:06.749444 10510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data/instance:
uuid: "b560308dae9e4c938a001df5fa9addad"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "c877a0e062bc9018db27634e9de3cfe4"
server_key_iv: "1cfec8147432cc0b1bf5a12f1b0666b9"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.749794 10510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal/instance:
uuid: "b560308dae9e4c938a001df5fa9addad"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "c877a0e062bc9018db27634e9de3cfe4"
server_key_iv: "1cfec8147432cc0b1bf5a12f1b0666b9"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.750923 10510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:47:06.751660 10525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.751813 10510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.751899 10510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal
uuid: "b560308dae9e4c938a001df5fa9addad"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "c877a0e062bc9018db27634e9de3cfe4"
server_key_iv: "1cfec8147432cc0b1bf5a12f1b0666b9"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.751961 10510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:06.760933 10510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:06.762598 10510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:06.762714 10510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:06.766179 10510 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.61:45657
I20260928 17:47:06.766223 10577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.61:45657 every 8 connection(s)
I20260928 17:47:06.766559 10510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
I20260928 17:47:06.767534 10578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:06.768908 10578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:47:06.772161 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 10510
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.61@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:06.777448 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.769531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.3.222.62:41825)
0928 17:47:06.769590 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.769592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.769748 (+ 156us) server_negotiation.cc:407] Connection header received
0928 17:47:06.770205 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.770206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.770218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.770263 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.770830 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.771366 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.772338 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.772625 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.772684 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.774836 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.774838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.774838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.774846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.776221 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.776559 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.776560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.776561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.776621 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.776822 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.776822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.776822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.777002 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.777068 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.777327 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.777387 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.222.61@KRBTEST.COM
I20260928 17:47:06.777760 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.769486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60204)
0928 17:47:06.769643 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:06.769678 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:06.770117 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:06.770336 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:06.770340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:06.770576 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:06.770748 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.770752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.771434 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.771435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.772215 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.772218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.772708 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.772709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.772857 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.773308 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:06.773316 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:06.774762 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:06.776285 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.776288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.776290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.776496 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.776685 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.776685 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.776685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.776774 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.777064 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:06.777067 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:06.777199 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 17:47:06.777305 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 17:47:06.777447 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260928 17:47:06.778319 10578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:47:06.780542 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.772636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.3.222.61:45657)
0928 17:47:06.772772 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.772776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.772897 (+ 121us) server_negotiation.cc:407] Connection header received
0928 17:47:06.772979 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.772981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.772995 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.773057 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.773599 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.774513 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.775397 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.775767 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.775817 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.778163 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.778169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.778169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.778175 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.779585 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.779867 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.779870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.779870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.779922 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.780074 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.780075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.780076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.780202 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.780345 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.780384 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.780450 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.61@KRBTEST.COM for oryx/127.3.222.61@KRBTEST.COM
I20260928 17:47:06.784243 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.778529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.0.0.1:58944)
0928 17:47:06.778567 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:06.778576 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:06.778606 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:06.778780 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:06.778780 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:06.778913 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:06.778973 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.778974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.779611 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.779611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.780062 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.780064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.780388 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.780389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.780443 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.780754 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:06.780757 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:06.781784 (+ 1027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:06.783429 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.783430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.783430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.783570 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.783717 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.783718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.783718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.783766 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.783953 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:06.783954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:06.783987 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 17:47:06.784044 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:47:06.784125 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 17:47:06.784322 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.778566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58944 (local address 127.3.222.61:45657)
0928 17:47:06.778690 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.778691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.778696 (+ 5us) server_negotiation.cc:407] Connection header received
0928 17:47:06.778713 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.778714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.778720 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.778772 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.779033 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.779544 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.780120 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.780336 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.780393 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.781855 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.781856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.781857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.781862 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.783364 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.783621 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.783622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.783622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.783670 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.783812 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.783813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.783813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.783906 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.783956 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.784068 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.784171 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260928 17:47:06.785429 10578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Bootstrap starting.
I20260928 17:47:06.786020 10578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:06.786298 10578 log.cc:824] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:06.787266 10578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: No bootstrap required, opened a new log
I20260928 17:47:06.788664 10578 raft_consensus.cc:348] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } }
I20260928 17:47:06.788805 10578 raft_consensus.cc:374] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:06.788848 10578 raft_consensus.cc:729] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b560308dae9e4c938a001df5fa9addad, State: Initialized, Role: FOLLOWER
I20260928 17:47:06.788959 10578 consensus_queue.cc:260] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } }
I20260928 17:47:06.789161 10585 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } } }
I20260928 17:47:06.789206 10578 sys_catalog.cc:564] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:06.789247 10585 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:47:06.790695 10596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:47:06.790746 10596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:47:06.792112 10510 master_runner.cc:428] Detected that this master b560308dae9e4c938a001df5fa9addad is joining an existing cluster
I20260928 17:47:06.792148 10510 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.222.61:45657
I20260928 17:47:06.798636 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.792984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.0.0.1:58950)
0928 17:47:06.793173 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:06.793182 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:06.793236 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:06.793364 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:06.793364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:06.793577 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:06.793675 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.793677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.794244 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.794244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.794901 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.794903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.795414 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.795414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.795479 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.795836 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:06.795840 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:06.796171 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:06.797696 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.797697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.797697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.797871 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.798037 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.798037 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.798037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.798083 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.798307 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:06.798308 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:06.798339 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 17:47:06.798400 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 17:47:06.798530 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260928 17:47:06.798687 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.793117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.3.222.61:45657)
0928 17:47:06.793163 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.793164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.793224 (+ 60us) server_negotiation.cc:407] Connection header received
0928 17:47:06.793280 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.793280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.793290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.793333 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.793748 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.794192 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.794978 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.795303 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.795339 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.796238 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.796239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.796239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.796245 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.797625 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.797936 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.797936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.797937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.797978 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.798138 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.798139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.798139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.798238 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.798272 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.798434 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.798592 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":30}
I20260928 17:47:06.799546 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.793017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.3.222.62:41825)
0928 17:47:06.793056 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.793058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.793237 (+ 179us) server_negotiation.cc:407] Connection header received
0928 17:47:06.793336 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.793337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.793348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.793398 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.793743 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.794497 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.795501 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.795787 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.795852 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.796862 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.796863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.796863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.796869 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.798215 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.798631 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.798633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.798633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.798699 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.799085 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.799086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.799086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.799237 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.799294 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.799433 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.799489 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:47:06.799608 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.793074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60212)
0928 17:47:06.793185 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:06.793193 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:06.793258 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:06.793406 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:06.793407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:06.793579 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:06.793675 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.793677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.794573 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.794574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.795422 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.795423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.795852 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.795853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.795911 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.796285 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:06.796289 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:06.796587 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:06.798285 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.798286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.798286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.798544 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.798837 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.798837 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.798838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.799027 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.799296 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:06.799297 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:06.799347 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 17:47:06.799416 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 17:47:06.799514 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":5,"thread_start_us":63,"threads_started":1}
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.62@KRBTEST.COM for oryx/127.3.222.61@KRBTEST.COM
I20260928 17:47:06.807355 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.801497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.3.222.61:45657)
0928 17:47:06.801541 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.801543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.801550 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:06.801603 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.801604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.801612 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.801651 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.802025 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.802480 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.803169 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.803391 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.803448 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.805287 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.805288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.805288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.805294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.806379 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.806691 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.806692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.806692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.806739 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.806887 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.806888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.806888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.806979 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.807029 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.807190 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.807277 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 17:47:06.807384 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.801347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.0.0.1:58952)
0928 17:47:06.801457 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:06.801481 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:06.801534 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:06.801665 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:06.801669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:06.801905 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:06.801961 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.801962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.802542 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.802543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.803103 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.803105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.803447 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.803448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.803529 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.803885 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:06.803889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:06.805218 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:06.806443 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.806449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.806451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.806633 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.806789 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.806789 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.806790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.806843 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.807027 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:06.807033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:06.807080 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 17:47:06.807148 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:47:06.807283 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
I20260928 17:47:06.807997 10046 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.222.61:45657: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b560308dae9e4c938a001df5fa9addad"
member_type: NON_VOTER
last_known_addr {
host: "127.3.222.61"
port: 45657
}
attrs {
promote: true
}
}
dest_uuid: "fdf0798ded714fc493705c56ef52d4cf"
cas_config_opid_index: -1
I20260928 17:47:06.808255 10046 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } }
I20260928 17:47:06.808668 10605 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b560308dae9e4c938a001df5fa9addad (127.3.222.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } }
I20260928 17:47:06.808930 10552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:06.809638 10605 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.222.61:45657
I20260928 17:47:06.809695 10552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Refusing update from remote peer fdf0798ded714fc493705c56ef52d4cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 17:47:06.809713 10605 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } }
I20260928 17:47:06.809765 10605 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:06.809835 10510 master.cc:537] Master@127.3.222.61:45657 shutting down...
I20260928 17:47:06.810325 10606 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 17:47:06.811455 10030 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 17:47:06.811810 10030 consensus_peers.cc:487] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf -> Peer b560308dae9e4c938a001df5fa9addad (127.3.222.61:45657): Couldn't send request to peer b560308dae9e4c938a001df5fa9addad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 17:47:06.812230 10510 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:06.812309 10510 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:06.812353 10510 tablet_replica.cc:330] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: stopping tablet replica
I20260928 17:47:06.824880 10510 master.cc:552] Master@127.3.222.61:45657 shutdown complete.
I20260928 17:47:06.824929 10510 master_runner.cc:305] Clearing existing system tablet
I20260928 17:47:06.825048 10510 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:06.825091 10510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:06.825724 10510 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260928 17:47:06.826171 10610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.826295 10510 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:06.826345 10510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.826366 10510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal
uuid: "b560308dae9e4c938a001df5fa9addad"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "c877a0e062bc9018db27634e9de3cfe4"
server_key_iv: "1cfec8147432cc0b1bf5a12f1b0666b9"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.826941 10510 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:06.828353 10510 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 17:47:06.828404 10510 log.cc:1196] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 17:47:06.828557 10510 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Deleting consensus metadata
I20260928 17:47:06.828651 10510 master_runner.cc:315] Copying system tablet from 127.3.222.62:41825
I20260928 17:47:06.829165 10510 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: tablet copy: Beginning tablet copy session from remote peer at address 127.3.222.62:41825
I20260928 17:47:06.834486 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.829472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.3.222.62:41825)
0928 17:47:06.829510 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.829512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.829602 (+ 90us) server_negotiation.cc:407] Connection header received
0928 17:47:06.829665 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.829666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.829675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.829720 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.830006 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.830491 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.831170 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.831442 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.831500 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.832367 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.832369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.832369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.832375 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.833532 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.833875 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.833875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.833876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.833923 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.834086 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.834086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.834086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.834173 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.834216 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.834356 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.834407 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:47:06.834540 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.829418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60216)
0928 17:47:06.829533 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:06.829543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:06.829606 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:06.829723 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:06.829723 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:06.829880 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:06.829942 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.829944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.830553 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.830554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.831103 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:06.831104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.831510 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.831511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:06.831581 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.831935 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:06.831940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:06.832294 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:06.833598 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.833599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.833600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.833805 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.833978 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:06.833978 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:06.833979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:06.834038 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:06.834229 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:06.834229 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:06.834272 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 17:47:06.834338 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 17:47:06.834427 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
I20260928 17:47:06.834842 10066 tablet_copy_service.cc:140] P fdf0798ded714fc493705c56ef52d4cf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b560308dae9e4c938a001df5fa9addad ({username='oryx', principal='oryx/127.3.222.61@KRBTEST.COM'} at 127.0.0.1:60216)
I20260928 17:47:06.834940 10066 tablet_copy_service.cc:161] P fdf0798ded714fc493705c56ef52d4cf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b560308dae9e4c938a001df5fa9addad at {username='oryx', principal='oryx/127.3.222.61@KRBTEST.COM'} at 127.0.0.1:60216: session id = b560308dae9e4c938a001df5fa9addad-00000000000000000000000000000000
I20260928 17:47:06.835748 10066 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 17:47:06.836277 10510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 17:47:06.837064 10510 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: tablet copy: Starting download of 0 data blocks...
I20260928 17:47:06.837183 10510 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: tablet copy: Starting download of 1 WAL segments...
I20260928 17:47:06.837934 10510 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 17:47:06.838595 10066 tablet_copy_service.cc:342] P fdf0798ded714fc493705c56ef52d4cf: Request end of tablet copy session b560308dae9e4c938a001df5fa9addad-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.222.61@KRBTEST.COM'} at 127.0.0.1:60216
I20260928 17:47:06.838639 10066 tablet_copy_service.cc:433] P fdf0798ded714fc493705c56ef52d4cf: ending tablet copy session b560308dae9e4c938a001df5fa9addad-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b560308dae9e4c938a001df5fa9addad
I20260928 17:47:06.840767 10510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:06.841763 10618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.841938 10510 server_base.cc:1034] running on GCE node
W20260928 17:47:06.842051 10620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:06.843303 10617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.843428 10510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:06.843559 10510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:06.844678 10510 hybrid_clock.cc:648] HybridClock initialized: now 1790617626844667 us; error 24 us; skew 500 ppm
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:06.845831 10510 init.cc:377] Logged in from keytab as oryx/127.3.222.61@KRBTEST.COM (short username oryx)
I20260928 17:47:06.846060 10510 webserver.cc:460] Webserver started at http://127.3.222.61:35929/ using document root <none> and password file <none>
I20260928 17:47:06.846163 10510 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:06.846195 10510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:06.846803 10510 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:06.847272 10627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.847378 10510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:06.847419 10510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal
uuid: "b560308dae9e4c938a001df5fa9addad"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "c877a0e062bc9018db27634e9de3cfe4"
server_key_iv: "1cfec8147432cc0b1bf5a12f1b0666b9"
server_key_version: "encryptionkey@0"
I20260928 17:47:06.847462 10510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:06.856336 10510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:06.857642 10510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:06.860028 10510 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.61:45657
I20260928 17:47:06.860302 10679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.61:45657 every 8 connection(s)
I20260928 17:47:06.860364 10510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
I20260928 17:47:06.861119 10680 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:47:06.861642 10680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Bootstrap starting.
I20260928 17:47:06.864338 10680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:47:06.864569 10680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Bootstrap complete.
I20260928 17:47:06.864755 10680 raft_consensus.cc:348] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } }
I20260928 17:47:06.864838 10680 raft_consensus.cc:729] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b560308dae9e4c938a001df5fa9addad, State: Initialized, Role: LEARNER
I20260928 17:47:06.864876 10680 consensus_queue.cc:260] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } }
I20260928 17:47:06.865039 10680 sys_catalog.cc:564] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:06.865039 10683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } }
I20260928 17:47:06.865140 10683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: This master's current role is: LEARNER
I20260928 17:47:06.866518 10694 catalog_manager.cc:1259] Loaded cluster ID: 924fdf473e524903a4fe69f6e2ebc043
I20260928 17:47:06.866561 10694 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: loading cluster ID for follower catalog manager: success
I20260928 17:47:06.867447 10694 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: acquiring CA information for follower catalog manager: success
I20260928 17:47:06.867722 10694 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:47:06.887964 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:06.882023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.3.222.61:45657)
0928 17:47:06.882145 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:06.882147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:06.882153 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:47:06.882172 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:06.882173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:06.882180 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:06.882234 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:06.882758 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.883429 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.884102 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:06.884288 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:06.884342 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:06.885550 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:06.885551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:06.885552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:06.885558 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:06.886881 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.887189 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.887190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.887190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.887282 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:06.887487 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:06.887488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:06.887488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:06.887590 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:06.887674 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:06.887771 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 17:47:06.887845 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.3.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.60.keytab.
Entry for principal oryx/127.3.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.60.keytab.
I20260928 17:47:06.911319 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.60:42451
--webserver_interface=127.3.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.60.keytab
--principal=oryx/127.3.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41825,127.3.222.61:45657,127.3.222.60:42451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:06.988860 10698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:06.989053 10698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:06.989078 10698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:06.990201 10698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:06.990247 10698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:06.990278 10698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:06.990295 10698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:06.990319 10698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:06.991796 10698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41825,127.3.222.61:45657,127.3.222.60:42451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.60.keytab
--principal=oryx/127.3.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.60:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:06.992018 10698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:06.992203 10698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:06.996462 10704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.996570 10698 server_base.cc:1034] running on GCE node
W20260928 17:47:06.996673 10706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:06.996682 10703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:06.996970 10698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:06.997190 10698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:06.998327 10698 hybrid_clock.cc:648] HybridClock initialized: now 1790617626998311 us; error 31 us; skew 500 ppm
Sep 28 17:47:06 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:07.000519 10698 init.cc:377] Logged in from keytab as oryx/127.3.222.60@KRBTEST.COM (short username oryx)
I20260928 17:47:07.000897 10698 webserver.cc:460] Webserver started at http://127.3.222.60:44725/ using document root <none> and password file <none>
I20260928 17:47:07.001055 10698 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:07.001091 10698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:07.001173 10698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:07.002066 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data/instance:
uuid: "7c61559e058948049e8f4442540799e2"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "14a23f960cf4729ee090c6d1e25825ad"
server_key_iv: "8dc3113da03fc11625e9ccb488c6e70a"
server_key_version: "encryptionkey@0"
I20260928 17:47:07.002341 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal/instance:
uuid: "7c61559e058948049e8f4442540799e2"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "14a23f960cf4729ee090c6d1e25825ad"
server_key_iv: "8dc3113da03fc11625e9ccb488c6e70a"
server_key_version: "encryptionkey@0"
I20260928 17:47:07.003463 10698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:47:07.004045 10713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.004191 10698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.004272 10698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal
uuid: "7c61559e058948049e8f4442540799e2"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "14a23f960cf4729ee090c6d1e25825ad"
server_key_iv: "8dc3113da03fc11625e9ccb488c6e70a"
server_key_version: "encryptionkey@0"
I20260928 17:47:07.004338 10698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:07.010634 10698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:07.012344 10698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:07.012475 10698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:07.015995 10698 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.60:42451
I20260928 17:47:07.016026 10765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.60:42451 every 8 connection(s)
I20260928 17:47:07.016304 10698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
I20260928 17:47:07.017138 10766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:07.018476 10766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.60@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:07.025440 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 10698
I20260928 17:47:07.027496 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.019154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.3.222.62:41825)
0928 17:47:07.019229 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.019231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.019317 (+ 86us) server_negotiation.cc:407] Connection header received
0928 17:47:07.019830 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.019831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.019842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.019891 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.020521 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.021008 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.022010 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.022197 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.022274 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.024303 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.024305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.024305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.024311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.026187 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.026605 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.026606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.026606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.026656 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.026905 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.026906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.026906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.027032 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.027095 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.027324 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.027402 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:47:07.027980 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.019020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60232)
0928 17:47:07.019185 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.019247 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.019752 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.019991 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.019994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.020233 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.020445 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.020449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.021076 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.021077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.021897 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.021900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.022309 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.022310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.022420 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.022786 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.022794 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.024216 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.026249 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.026255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.026257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.026533 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.026724 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.026725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.026725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.026851 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.027087 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.027090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.027188 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.027291 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.027835 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":375,"thread_start_us":77,"threads_started":1}
I20260928 17:47:07.028432 10766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.222.60@KRBTEST.COM
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.60@KRBTEST.COM for oryx/127.3.222.61@KRBTEST.COM
I20260928 17:47:07.033493 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.025895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.3.222.60:42451)
0928 17:47:07.026023 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.026028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.026079 (+ 51us) server_negotiation.cc:407] Connection header received
0928 17:47:07.026106 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.026108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.026121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.026189 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.026791 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.027720 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.028581 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.028934 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.028990 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.031087 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.031094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.031095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.031101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.032535 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.032821 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.032823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.032824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.032874 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.033027 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.033028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.033028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.033152 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.033296 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.033338 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.033404 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260928 17:47:07.034421 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.028658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.0.0.1:58966)
0928 17:47:07.028695 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.028704 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.028732 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.028872 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.028873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.028974 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.029013 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.029015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.029736 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.029737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.030215 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.030217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.030540 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.030540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.030601 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.030987 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.030990 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.032225 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.033547 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.033548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.033548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.033723 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.033917 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.033918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.033918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.033975 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.034226 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.034226 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.034258 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.034309 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.034369 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:47:07.034451 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.028717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.3.222.61:45657)
0928 17:47:07.028772 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.028774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.028782 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:07.028802 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.028803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.028822 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.028864 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.029076 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.029650 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.030280 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.030487 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.030536 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.032297 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.032298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.032298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.032304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.033481 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.033788 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.033789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.033789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.033862 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.034029 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.034030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.034030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.034126 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.034182 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.034331 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.034392 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:47:07.034865 10766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.60@KRBTEST.COM for oryx/127.3.222.60@KRBTEST.COM
I20260928 17:47:07.040544 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.035055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42451 (local address 127.0.0.1:53034)
0928 17:47:07.035093 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.035102 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.035136 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.035270 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.035270 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.035367 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.035392 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.035393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.035926 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.035927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.036396 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.036397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.036735 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.036735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.036779 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.037059 (+ 280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.037063 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.038170 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.039723 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.039724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.039724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.039876 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.040023 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.040024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.040024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.040073 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.040258 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.040259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.040291 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.040349 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.040433 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 17:47:07.040588 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.035093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.3.222.60:42451)
0928 17:47:07.035122 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.035124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.035131 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:07.035171 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.035171 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.035179 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.035259 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.035440 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.035862 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.036465 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.036648 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.036710 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.038243 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.038244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.038244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.038250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.039664 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.039930 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.039931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.039932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.039976 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.040119 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.040120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.040120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.040211 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.040260 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.040374 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.040477 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":14}
I20260928 17:47:07.041708 10766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: Bootstrap starting.
I20260928 17:47:07.042408 10766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:07.042615 10766 log.cc:824] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:07.043196 10766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: No bootstrap required, opened a new log
I20260928 17:47:07.044927 10766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } }
I20260928 17:47:07.045037 10766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:07.045058 10766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c61559e058948049e8f4442540799e2, State: Initialized, Role: FOLLOWER
I20260928 17:47:07.045141 10766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } }
I20260928 17:47:07.045353 10766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:07.045336 10771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } } }
I20260928 17:47:07.045444 10771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:47:07.046794 10782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:47:07.046851 10782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:47:07.048753 10698 master_runner.cc:428] Detected that this master 7c61559e058948049e8f4442540799e2 is joining an existing cluster
I20260928 17:47:07.048799 10698 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.222.60:42451
I20260928 17:47:07.055928 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.049965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53038 (local address 127.3.222.60:42451)
0928 17:47:07.050018 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.050020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.050186 (+ 166us) server_negotiation.cc:407] Connection header received
0928 17:47:07.050256 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.050256 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.050262 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.050286 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.050675 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.051410 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.052502 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.052768 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.052803 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.053851 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.053852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.053852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.053858 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.054990 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.055362 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.055363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.055363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.055405 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.055558 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.055559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.055559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.055635 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.055674 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.055814 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.055876 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:47:07.056033 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.049939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42451 (local address 127.0.0.1:53038)
0928 17:47:07.050146 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.050154 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.050202 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.050340 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.050340 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.050547 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.050602 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.050604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.051571 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.051572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.052418 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.052421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.052863 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.052864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.052915 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.053363 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.053368 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.053767 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.055060 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.055061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.055061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.055288 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.055446 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.055447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.055447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.055506 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.055682 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.055683 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.055732 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.055800 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.055936 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":74,"thread_start_us":61,"threads_started":1}
I20260928 17:47:07.066382 10771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:07.066495 10771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } }
I20260928 17:47:07.066727 10771 leader_election.cc:288] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdf0798ded714fc493705c56ef52d4cf (127.3.222.62:41825), b560308dae9e4c938a001df5fa9addad (127.3.222.61:45657)
I20260928 17:47:07.066927 10056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c61559e058948049e8f4442540799e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf0798ded714fc493705c56ef52d4cf" is_pre_election: true
I20260928 17:47:07.067168 10654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c61559e058948049e8f4442540799e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b560308dae9e4c938a001df5fa9addad" is_pre_election: true
I20260928 17:47:07.067308 10654 raft_consensus.cc:1799] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 LEARNER]: Handling vote request from an unknown peer 7c61559e058948049e8f4442540799e2
I20260928 17:47:07.067363 10654 raft_consensus.cc:2396] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 7c61559e058948049e8f4442540799e2 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260928 17:47:07.067539 10714 leader_election.cc:302] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c61559e058948049e8f4442540799e2; no voters: b560308dae9e4c938a001df5fa9addad, fdf0798ded714fc493705c56ef52d4cf
I20260928 17:47:07.067693 10771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:07.068612 10771 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 17:47:07.070554 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.049708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60242)
0928 17:47:07.050047 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.050056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.050110 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.050331 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.050332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.050518 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.050587 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.050589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.051489 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.051490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.052447 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.052450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.052951 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.052952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.053014 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.067600 (+ 14586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.067605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.067973 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.069566 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.069567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.069567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.069754 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.069956 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.069956 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.069957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.070013 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.070261 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.070261 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.070305 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.070372 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.070455 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":94,"thread_start_us":89,"threads_started":1}
I20260928 17:47:07.070566 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.049773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.3.222.62:41825)
0928 17:47:07.049820 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.049822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.050218 (+ 396us) server_negotiation.cc:407] Connection header received
0928 17:47:07.050239 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.050240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.050250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.050292 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.050664 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.051414 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.052540 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.052831 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.052874 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.068072 (+ 15198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.068074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.068074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.068082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.069501 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.069829 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.069831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.069831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.069890 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.070072 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.070073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.070073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.070201 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.070264 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.070404 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.070479 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:47:07.071465 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.049858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.3.222.61:45657)
0928 17:47:07.049893 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.049894 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.050032 (+ 138us) server_negotiation.cc:407] Connection header received
0928 17:47:07.050126 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.050127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.050138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.050186 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.050537 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.051347 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.052539 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.052831 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.052868 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.068949 (+ 16081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.068950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.068951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.068957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.070375 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.070743 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.070745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.070745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.070803 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.070976 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.070977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.070977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.071085 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.071116 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.071276 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.071353 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:07.071617 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.049706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.0.0.1:58980)
0928 17:47:07.049949 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.049959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.050061 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.050188 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.050189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.050388 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.050453 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.050455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.051451 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.051452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.052447 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.052450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.052950 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.052950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.053014 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.068531 (+ 15517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.068536 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.068874 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.070445 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.070446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.070446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.070666 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.070871 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.070871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.070872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.070920 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.071147 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.071148 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.071176 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.071253 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.071512 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":92,"spinlock_wait_cycles":9088,"thread_start_us":99,"threads_started":1}
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617626, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.62@KRBTEST.COM for oryx/127.3.222.60@KRBTEST.COM
I20260928 17:47:07.077910 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.072428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.3.222.60:42451)
0928 17:47:07.072463 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.072465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.072473 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:07.072529 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.072530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.072538 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.072580 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.072800 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.073226 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.073848 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.074037 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.074087 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.075880 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.075881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.075882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.075887 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.077023 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.077369 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.077370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.077370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.077409 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.077555 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.077556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.077556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.077634 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.077674 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.077814 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.077861 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:07.077977 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.072380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42451 (local address 127.0.0.1:53046)
0928 17:47:07.072425 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.072435 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.072470 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.072581 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.072582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.072710 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.072744 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.072746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.073285 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.073286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.073778 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.073779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.074090 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.074091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.074151 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.074467 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.074471 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.075797 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.077095 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.077095 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.077096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.077291 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.077453 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.077454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.077454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.077509 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.077681 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.077682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.077723 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.077789 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.077882 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 17:47:07.078408 10046 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.222.60:42451: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7c61559e058948049e8f4442540799e2"
member_type: NON_VOTER
last_known_addr {
host: "127.3.222.60"
port: 42451
}
attrs {
promote: true
}
}
dest_uuid: "fdf0798ded714fc493705c56ef52d4cf"
cas_config_opid_index: 5
I20260928 17:47:07.078634 10046 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } }
I20260928 17:47:07.078962 10606 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7c61559e058948049e8f4442540799e2 (127.3.222.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.079157 10740 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 FOLLOWER]: Refusing update from remote peer fdf0798ded714fc493705c56ef52d4cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 17:47:07.079392 10605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:07.079936 10606 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.222.60:42451
I20260928 17:47:07.079994 10606 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.080049 10606 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:07.080278 10698 master.cc:537] Master@127.3.222.60:42451 shutting down...
I20260928 17:47:07.080626 10740 raft_consensus.cc:858] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } }
W20260928 17:47:07.080672 10740 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } }
I20260928 17:47:07.081087 10740 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b560308dae9e4c938a001df5fa9addad (127.3.222.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } }
I20260928 17:47:07.082746 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.078994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.0.0.1:58992)
0928 17:47:07.079106 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.079116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.079150 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.079498 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.079499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.079501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:07.079534 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.079535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.080402 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.080403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.081562 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.081565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.082505 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.082506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.082571 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.082593 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.082654 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.082688 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 17:47:07.082752 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.079076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.3.222.61:45657)
0928 17:47:07.079345 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.079347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.079354 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:07.079372 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.079373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.079392 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.079448 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:07.079595 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.080306 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.081654 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.082420 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.082484 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.082556 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.082662 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.082691 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260928 17:47:07.083016 10791 mvcc.cc:205] Tried to move back new op lower bound from 7334369796663398400 to 7334369796557733888. Current Snapshot: MvccSnapshot[applied={T|T < 7334369796567162880}]
I20260928 17:47:07.083248 10654 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 LEARNER]: Refusing update from remote peer fdf0798ded714fc493705c56ef52d4cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 17:47:07.083465 10605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 17:47:07.082999 10771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf0798ded714fc493705c56ef52d4cf. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } }
I20260928 17:47:07.083813 10771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 17:47:07.083851 10771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } }
I20260928 17:47:07.083889 10771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 17:47:07.083865 10793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } }
I20260928 17:47:07.083940 10793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 17:47:07.084012 10654 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7c61559e058948049e8f4442540799e2 (127.3.222.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.085060 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf0798ded714fc493705c56ef52d4cf. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.085222 10683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.085330 10683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: This master's current role is: LEARNER
I20260928 17:47:07.085304 10605 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: NON_VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: true } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.085366 10605 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:07.085834 10698 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260928 17:47:07.085873 10032 consensus_peers.cc:487] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf -> Peer 7c61559e058948049e8f4442540799e2 (127.3.222.60:42451): Couldn't send request to peer 7c61559e058948049e8f4442540799e2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 17:47:07.085932 10698 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 17:47:07.085963 10698 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: stopping tablet replica
I20260928 17:47:07.085974 10606 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: attempting to promote NON_VOTER b560308dae9e4c938a001df5fa9addad to VOTER
I20260928 17:47:07.086135 10606 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } }
I20260928 17:47:07.086659 10654 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 LEARNER]: Refusing update from remote peer fdf0798ded714fc493705c56ef52d4cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260928 17:47:07.086707 10032 consensus_peers.cc:487] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf -> Peer 7c61559e058948049e8f4442540799e2 (127.3.222.60:42451): Couldn't send request to peer 7c61559e058948049e8f4442540799e2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 17:47:07.086846 10605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 17:47:07.085208 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: This master's current role is: LEARNER
I20260928 17:47:07.087987 10605 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b560308dae9e4c938a001df5fa9addad (127.3.222.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
W20260928 17:47:07.088217 9999 connection.cc:535] client connection to 127.3.222.60:42451 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:47:07.088799 10605 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.088869 10605 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:07.088907 10605 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.088929 10605 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:07.089174 10654 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b560308dae9e4c938a001df5fa9addad (127.3.222.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.089947 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.090025 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:47:07.100710 10698 master.cc:552] Master@127.3.222.60:42451 shutdown complete.
I20260928 17:47:07.100747 10698 master_runner.cc:305] Clearing existing system tablet
I20260928 17:47:07.100854 10698 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:07.100890 10698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:07.101382 10698 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:07.101828 10802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.101964 10698 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:07.102015 10698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.102036 10698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal
uuid: "7c61559e058948049e8f4442540799e2"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "14a23f960cf4729ee090c6d1e25825ad"
server_key_iv: "8dc3113da03fc11625e9ccb488c6e70a"
server_key_version: "encryptionkey@0"
I20260928 17:47:07.102587 10698 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:07.103765 10698 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 17:47:07.103818 10698 log.cc:1196] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 17:47:07.104024 10698 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: Deleting consensus metadata
I20260928 17:47:07.104127 10698 master_runner.cc:315] Copying system tablet from 127.3.222.62:41825
I20260928 17:47:07.104712 10698 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: tablet copy: Beginning tablet copy session from remote peer at address 127.3.222.62:41825
I20260928 17:47:07.110594 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.104986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60244)
0928 17:47:07.105106 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.105116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.105172 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.105299 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.105300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.105489 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.105533 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.105536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.106460 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.106461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.106975 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.106976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.107463 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.107464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.107506 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.107919 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.107923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.108277 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.109785 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.109786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.109786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.109937 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.110117 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.110118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.110118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.110159 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.110379 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.110380 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.110414 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.110460 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.110534 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260928 17:47:07.110637 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.105033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.3.222.62:41825)
0928 17:47:07.105070 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.105072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.105170 (+ 98us) server_negotiation.cc:407] Connection header received
0928 17:47:07.105220 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.105221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.105232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.105293 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.105610 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.106382 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.107045 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.107397 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.107465 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.108375 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.108376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.108376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.108383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.109717 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.109997 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.109998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.109998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.110059 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.110193 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.110194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.110194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.110322 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.110384 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.110486 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.110557 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:07.110790 10066 tablet_copy_service.cc:140] P fdf0798ded714fc493705c56ef52d4cf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7c61559e058948049e8f4442540799e2 ({username='oryx', principal='oryx/127.3.222.60@KRBTEST.COM'} at 127.0.0.1:60244)
I20260928 17:47:07.110849 10066 tablet_copy_service.cc:161] P fdf0798ded714fc493705c56ef52d4cf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7c61559e058948049e8f4442540799e2 at {username='oryx', principal='oryx/127.3.222.60@KRBTEST.COM'} at 127.0.0.1:60244: session id = 7c61559e058948049e8f4442540799e2-00000000000000000000000000000000
I20260928 17:47:07.111305 10066 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 17:47:07.111681 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 17:47:07.112504 10698 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: tablet copy: Starting download of 0 data blocks...
I20260928 17:47:07.112607 10698 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: tablet copy: Starting download of 1 WAL segments...
I20260928 17:47:07.113219 10698 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 17:47:07.113910 10066 tablet_copy_service.cc:342] P fdf0798ded714fc493705c56ef52d4cf: Request end of tablet copy session 7c61559e058948049e8f4442540799e2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.222.60@KRBTEST.COM'} at 127.0.0.1:60244
I20260928 17:47:07.113956 10066 tablet_copy_service.cc:433] P fdf0798ded714fc493705c56ef52d4cf: ending tablet copy session 7c61559e058948049e8f4442540799e2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7c61559e058948049e8f4442540799e2
I20260928 17:47:07.115897 10698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:07.116982 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:07.116981 10809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:07.124346 10698 server_base.cc:1034] running on GCE node
W20260928 17:47:07.124364 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:07.124583 10698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:07.124706 10698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:07.125823 10698 hybrid_clock.cc:648] HybridClock initialized: now 1790617627125811 us; error 27 us; skew 500 ppm
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:07.127112 10698 init.cc:377] Logged in from keytab as oryx/127.3.222.60@KRBTEST.COM (short username oryx)
I20260928 17:47:07.127377 10698 webserver.cc:460] Webserver started at http://127.3.222.60:44929/ using document root <none> and password file <none>
I20260928 17:47:07.127487 10698 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:07.127517 10698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:07.128037 10698 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:07.128472 10819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.128597 10698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.128638 10698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal
uuid: "7c61559e058948049e8f4442540799e2"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "14a23f960cf4729ee090c6d1e25825ad"
server_key_iv: "8dc3113da03fc11625e9ccb488c6e70a"
server_key_version: "encryptionkey@0"
I20260928 17:47:07.128690 10698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:07.135180 10698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:07.136547 10698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:07.139744 10698 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.60:42451
I20260928 17:47:07.139955 10871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.60:42451 every 8 connection(s)
I20260928 17:47:07.140050 10698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
I20260928 17:47:07.140894 10872 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:47:07.141419 10872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: Bootstrap starting.
I20260928 17:47:07.143604 10872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:47:07.143795 10872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: Bootstrap complete.
I20260928 17:47:07.143959 10872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } }
I20260928 17:47:07.144042 10872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c61559e058948049e8f4442540799e2, State: Initialized, Role: LEARNER
I20260928 17:47:07.144076 10872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } }
I20260928 17:47:07.144240 10872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:07.144225 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.144320 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: This master's current role is: LEARNER
I20260928 17:47:07.145627 10886 catalog_manager.cc:1259] Loaded cluster ID: 924fdf473e524903a4fe69f6e2ebc043
I20260928 17:47:07.145671 10886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: loading cluster ID for follower catalog manager: success
I20260928 17:47:07.146587 10886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: acquiring CA information for follower catalog manager: success
I20260928 17:47:07.146838 10886 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:47:07.195626 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.189190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.3.222.60:42451)
0928 17:47:07.189307 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.189309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.189317 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:07.189340 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.189341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.189349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.189401 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.190937 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.191593 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.192305 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.192497 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.192545 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.193673 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.193675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.193675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.193681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.194762 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.195030 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.195030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.195031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.195074 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.195233 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.195234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.195234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.195353 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.195425 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.195504 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.195560 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260928 17:47:07.196058 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 10108
I20260928 17:47:07.200140 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:36007
--local_ip_for_outbound_sockets=127.3.222.1
--tserver_master_addrs=127.3.222.62:41825
--webserver_port=45907
--webserver_interface=127.3.222.1
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab
--principal=oryx/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.222.62:41825,127.3.222.61:45657
--tserver_master_addrs=127.3.222.62:41825,127.3.222.61:45657,127.3.222.60:42451 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:07.277204 10888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:07.277372 10888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:07.277393 10888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:07.278476 10888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:07.278518 10888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:07.278545 10888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:47:07.279965 10888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab
--principal=oryx/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:36007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=45907
--tserver_master_addrs=127.3.222.62:41825,127.3.222.61:45657,127.3.222.60:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:07.280156 10888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:07.280360 10888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:07.284873 10893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:07.284963 10894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:07.284883 10896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:07.285064 10888 server_base.cc:1034] running on GCE node
I20260928 17:47:07.285233 10888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:07.285435 10888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:07.286584 10888 hybrid_clock.cc:648] HybridClock initialized: now 1790617627286569 us; error 24 us; skew 500 ppm
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:07.288863 10888 init.cc:377] Logged in from keytab as oryx/127.3.222.1@KRBTEST.COM (short username oryx)
I20260928 17:47:07.289301 10888 webserver.cc:460] Webserver started at http://127.3.222.1:45907/ using document root <none> and password file <none>
I20260928 17:47:07.289494 10888 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:07.289551 10888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:07.290761 10888 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:07.291282 10903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.291424 10888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.291500 10888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "47845caf59fc4a6f8253c8b6acd59276"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "8d2e43741bf09b87a876ce1b2cb58ad0"
server_key_iv: "35c7d2f8db680d6da33c5383f72c4f29"
server_key_version: "encryptionkey@0"
I20260928 17:47:07.291801 10888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:07.300086 10888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:07.301664 10888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:07.301782 10888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:07.302001 10888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:07.302263 10888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:07.302294 10888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.302320 10888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:07.302343 10888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.307669 10888 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:36007
I20260928 17:47:07.307710 11016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:36007 every 8 connection(s)
I20260928 17:47:07.308002 10888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:47:07.314123 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 10888
I20260928 17:47:07.314270 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 10242
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.61@KRBTEST.COM
I20260928 17:47:07.318065 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.308768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:38855 (local address 127.3.222.61:45657)
0928 17:47:07.308813 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.308814 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.309000 (+ 186us) server_negotiation.cc:407] Connection header received
0928 17:47:07.309503 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.309505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.309518 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.309564 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.310380 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.311141 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.312952 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.313138 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.313187 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.315661 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.315663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.315664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.315671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.316825 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.317271 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.317272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.317273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.317327 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.317555 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.317555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.317556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.317655 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.317699 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.317946 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.318004 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:47:07.318292 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.308661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.3.222.1:38855)
0928 17:47:07.308906 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.308923 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.309420 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.309892 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.309893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.310113 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.310295 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.310300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.311276 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.311277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.312445 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.312447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.313237 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.313238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.313298 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.313738 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.313742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.315559 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.316892 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.316898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.316901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.317175 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.317386 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.317386 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.317387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.317505 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.317701 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.317704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.317837 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.317933 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.318084 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":49,"thread_start_us":59,"threads_started":1}
I20260928 17:47:07.318766 11019 heartbeater.cc:347] Connected to a master server at 127.3.222.61:45657
I20260928 17:47:07.318892 11019 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:07.319082 11019 heartbeater.cc:511] Master 127.3.222.61:45657 requested a full tablet report, sending...
I20260928 17:47:07.319242 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:46439
--local_ip_for_outbound_sockets=127.3.222.2
--tserver_master_addrs=127.3.222.62:41825
--webserver_port=41145
--webserver_interface=127.3.222.2
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab
--principal=oryx/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.222.62:41825,127.3.222.61:45657
--tserver_master_addrs=127.3.222.62:41825,127.3.222.61:45657,127.3.222.60:42451 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.60@KRBTEST.COM
I20260928 17:47:07.320897 10644 ts_manager.cc:194] Registered new tserver with Master: 47845caf59fc4a6f8253c8b6acd59276 (127.3.222.1:36007)
I20260928 17:47:07.322678 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.308661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.1:46615)
0928 17:47:07.308878 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.308923 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.309411 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.309646 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.309651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.310106 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.310297 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.310303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.311788 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.311789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.312425 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.312427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.312850 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.312851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.312984 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.313601 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.313612 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.319927 (+ 6315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.321745 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.321745 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.321746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.321892 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.322074 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.322075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.322075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.322128 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.322369 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.322370 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.322416 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.322493 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.322583 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260928 17:47:07.322682 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.308913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:46615 (local address 127.3.222.62:41825)
0928 17:47:07.308967 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.308969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.308997 (+ 28us) server_negotiation.cc:407] Connection header received
0928 17:47:07.309527 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.309528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.309542 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.309572 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.311246 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.311719 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.312493 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.312782 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.312844 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.320023 (+ 7179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.320024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.320025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.320033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.321679 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.321964 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.321965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.321965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.322021 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.322175 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.322176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.322176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.322301 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.322368 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.322522 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.322597 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:47:07.322989 11018 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41825
I20260928 17:47:07.323033 11018 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:07.323099 11018 heartbeater.cc:511] Master 127.3.222.62:41825 requested a full tablet report, sending...
I20260928 17:47:07.323374 10046 ts_manager.cc:194] Re-registered known tserver with Master: 47845caf59fc4a6f8253c8b6acd59276 (127.3.222.1:36007)
I20260928 17:47:07.323447 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.308657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:59971 (local address 127.3.222.60:42451)
0928 17:47:07.308701 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.308703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.309100 (+ 397us) server_negotiation.cc:407] Connection header received
0928 17:47:07.309528 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.309530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.309543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.309572 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.310406 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.311131 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.312551 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.312849 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.312910 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.320752 (+ 7842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.320754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.320754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.320761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.322161 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.322453 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.322454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.322455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.322509 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.322694 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.322694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.322695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.322824 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.322864 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.323294 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.323371 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:47:07.323849 10046 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:46615
I20260928 17:47:07.323905 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.308661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42451 (local address 127.3.222.1:59971)
0928 17:47:07.309048 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.309056 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.309417 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.309651 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.309652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.310091 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.310297 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.310303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.311272 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.311273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.312433 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.312436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.312914 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.312914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.312984 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.313627 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.313631 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.320643 (+ 7012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.322224 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.322225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.322225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.322402 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.322575 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.322576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.322576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.322630 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.323089 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.323090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.323156 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.323263 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.323356 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":138,"thread_start_us":70,"threads_started":1}
I20260928 17:47:07.324481 11017 heartbeater.cc:347] Connected to a master server at 127.3.222.60:42451
I20260928 17:47:07.324522 11017 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:07.324559 11017 heartbeater.cc:511] Master 127.3.222.60:42451 requested a full tablet report, sending...
I20260928 17:47:07.324880 10836 ts_manager.cc:194] Registered new tserver with Master: 47845caf59fc4a6f8253c8b6acd59276 (127.3.222.1:36007)
W20260928 17:47:07.400171 11024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:07.400349 11024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:07.400369 11024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:07.401449 11024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:07.401486 11024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:07.401510 11024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:47:07.402908 11024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab
--principal=oryx/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:46439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=41145
--tserver_master_addrs=127.3.222.62:41825,127.3.222.61:45657,127.3.222.60:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:07.403111 11024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:07.403337 11024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:07.407929 11029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:07.407959 11030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:07.408010 11024 server_base.cc:1034] running on GCE node
W20260928 17:47:07.408650 11032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:07.408847 11024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:07.409065 11024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:07.410198 11024 hybrid_clock.cc:648] HybridClock initialized: now 1790617627410186 us; error 25 us; skew 500 ppm
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:07.412463 11024 init.cc:377] Logged in from keytab as oryx/127.3.222.2@KRBTEST.COM (short username oryx)
I20260928 17:47:07.412866 11024 webserver.cc:460] Webserver started at http://127.3.222.2:41145/ using document root <none> and password file <none>
I20260928 17:47:07.413105 11024 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:07.413164 11024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:07.414461 11024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:47:07.414940 11039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.415055 11024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.415124 11024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "ca5c6d1a25304fe79e860d08d35d55df"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "7a70a00d3ba4edc76890d45de46b93e3"
server_key_iv: "0eb0350dd18f145d0a501483e53d3958"
server_key_version: "encryptionkey@0"
I20260928 17:47:07.415473 11024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:07.443477 11024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:07.445509 11024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:07.445632 11024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:07.445845 11024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:07.446113 11024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:07.446154 11024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.446182 11024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:07.446197 11024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.451421 11024 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:46439
I20260928 17:47:07.451474 11152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:46439 every 8 connection(s)
I20260928 17:47:07.451759 11024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:47:07.454615 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 11024
I20260928 17:47:07.454756 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 10376
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.61@KRBTEST.COM
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.60@KRBTEST.COM
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:07.461186 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:35127
--local_ip_for_outbound_sockets=127.3.222.3
--tserver_master_addrs=127.3.222.62:41825
--webserver_port=36367
--webserver_interface=127.3.222.3
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab
--principal=oryx/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.222.62:41825,127.3.222.61:45657
--tserver_master_addrs=127.3.222.62:41825,127.3.222.61:45657,127.3.222.60:42451 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:47:07.462738 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.452433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:57897 (local address 127.3.222.62:41825)
0928 17:47:07.452484 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.452486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.452670 (+ 184us) server_negotiation.cc:407] Connection header received
0928 17:47:07.453358 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.453359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.453371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.453426 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.454149 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.454886 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.457060 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.457348 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.457392 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.459699 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.459700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.459701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.459708 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.461153 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.461573 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.461575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.461575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.461631 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.461891 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.461892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.461892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.462240 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.462293 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.462567 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.462651 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:47:07.462742 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.452585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:38431 (local address 127.3.222.61:45657)
0928 17:47:07.452632 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.452634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.452840 (+ 206us) server_negotiation.cc:407] Connection header received
0928 17:47:07.453245 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.453246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.453256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.453284 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.454112 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.454890 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.455936 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.456235 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.456292 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.459698 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.459699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.459700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.459706 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.461302 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.461830 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.461831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.461832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.461884 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.462045 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.462047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.462047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.462184 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.462240 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.462581 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.462655 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:47:07.462910 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.452372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.3.222.2:38431)
0928 17:47:07.452796 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.452804 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.453158 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.453666 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.453667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.453844 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.454029 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.454033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.454973 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.454974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.455857 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.455859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.456349 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.456350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.456481 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.456976 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.456986 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.459603 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.461543 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.461544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.461544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.461754 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.461947 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.461947 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.461948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.461995 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.462265 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.462271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.462418 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.462552 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.462699 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":81,"spinlock_wait_cycles":8064,"thread_start_us":100,"threads_started":1}
I20260928 17:47:07.463047 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.452372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.2:57897)
0928 17:47:07.452550 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.452591 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.453135 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.453435 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.453436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.453829 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.454030 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.454035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.454973 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.454974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.456585 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.456587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.457466 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.457467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.457533 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.457907 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.457911 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.459601 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.461219 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.461226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.461229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.461488 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.461701 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.461702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.461702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.461820 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.462314 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.462315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.462418 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.462879 (+ 461us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.462959 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":820,"thread_start_us":73,"threads_started":1}
I20260928 17:47:07.463363 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.452610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:37515 (local address 127.3.222.60:42451)
0928 17:47:07.452659 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.452660 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.452710 (+ 50us) server_negotiation.cc:407] Connection header received
0928 17:47:07.453230 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.453231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.453242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.453284 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.454127 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.454802 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.456061 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.456336 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.456388 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.459296 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.459298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.459298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.459305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.461823 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.462147 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.462148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.462148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.462199 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.462373 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.462374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.462374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.462489 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.462541 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.463172 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.463270 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:47:07.463495 11155 heartbeater.cc:347] Connected to a master server at 127.3.222.61:45657
I20260928 17:47:07.463589 11155 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:07.463593 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.452372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42451 (local address 127.3.222.2:37515)
0928 17:47:07.452656 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.452664 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.453136 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.453362 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.453366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.453844 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.454030 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.454035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.454911 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.454912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.455974 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.455976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.456437 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.456438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.456500 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.457034 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.457039 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.459171 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.461892 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.461892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.461893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.462075 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.462266 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.462266 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.462267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.462315 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.462564 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.462565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.462604 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.462670 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.462752 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":58,"thread_start_us":85,"threads_started":1}
I20260928 17:47:07.463770 11155 heartbeater.cc:511] Master 127.3.222.61:45657 requested a full tablet report, sending...
I20260928 17:47:07.463914 11153 heartbeater.cc:347] Connected to a master server at 127.3.222.60:42451
I20260928 17:47:07.463958 11153 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:07.464012 11153 heartbeater.cc:511] Master 127.3.222.60:42451 requested a full tablet report, sending...
I20260928 17:47:07.464080 11154 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41825
I20260928 17:47:07.464119 11154 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:07.464169 11154 heartbeater.cc:511] Master 127.3.222.62:41825 requested a full tablet report, sending...
I20260928 17:47:07.464206 10644 ts_manager.cc:194] Registered new tserver with Master: ca5c6d1a25304fe79e860d08d35d55df (127.3.222.2:46439)
I20260928 17:47:07.464224 10836 ts_manager.cc:194] Registered new tserver with Master: ca5c6d1a25304fe79e860d08d35d55df (127.3.222.2:46439)
I20260928 17:47:07.464352 10046 ts_manager.cc:194] Re-registered known tserver with Master: ca5c6d1a25304fe79e860d08d35d55df (127.3.222.2:46439)
I20260928 17:47:07.464735 10046 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:57897
W20260928 17:47:07.540949 11160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:07.541136 11160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:07.541157 11160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:07.542273 11160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:07.542313 11160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:07.542337 11160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:07.543776 11160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab
--principal=oryx/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:35127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=36367
--tserver_master_addrs=127.3.222.62:41825,127.3.222.61:45657,127.3.222.60:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:07.543975 11160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:07.544209 11160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:07.548831 11165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:07.548847 11166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:07.548831 11168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:07.549062 11160 server_base.cc:1034] running on GCE node
I20260928 17:47:07.549238 11160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:07.549414 11160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:07.550551 11160 hybrid_clock.cc:648] HybridClock initialized: now 1790617627550533 us; error 27 us; skew 500 ppm
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:07.552860 11160 init.cc:377] Logged in from keytab as oryx/127.3.222.3@KRBTEST.COM (short username oryx)
I20260928 17:47:07.553269 11160 webserver.cc:460] Webserver started at http://127.3.222.3:36367/ using document root <none> and password file <none>
I20260928 17:47:07.553458 11160 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:07.553519 11160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:07.554858 11160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:07.555492 11175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.555634 11160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.555711 11160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "f45792f6552b400b817ec4742a71836a"
format_stamp: "Formatted at 2026-09-28 17:47:06 on dist-test-slave-w3bm"
server_key: "209819d69f51c291dce4207ceae9cac7"
server_key_iv: "ba59946f2834f2f0fcb52cf95ca66005"
server_key_version: "encryptionkey@0"
I20260928 17:47:07.556006 11160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:07.564993 11160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:07.566875 11160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:07.566974 11160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:07.567142 11160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:07.567428 11160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:07.567459 11160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.567479 11160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:07.567502 11160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:07.572892 11160 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:35127
I20260928 17:47:07.572950 11288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:35127 every 8 connection(s)
I20260928 17:47:07.573242 11160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:47:07.577823 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 11160
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.60@KRBTEST.COM
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.61@KRBTEST.COM
Sep 28 17:47:07 dist-test-slave-w3bm krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617627, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:07.584203 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.573912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.3:41777)
0928 17:47:07.574385 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.574393 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.574644 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.574848 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.574852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.575296 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.575482 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.575487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.577197 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.577198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.577982 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.577985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.578476 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.578477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.578545 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.578840 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.578844 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.581095 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.582802 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.582803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.582803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.583021 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.583228 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.583228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.583229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.583282 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.583551 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.583557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.583717 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.583814 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.583965 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":20,"thread_start_us":79,"threads_started":1}
I20260928 17:47:07.584316 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.574202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:45195 (local address 127.3.222.61:45657)
0928 17:47:07.574243 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.574245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.574830 (+ 585us) server_negotiation.cc:407] Connection header received
0928 17:47:07.574881 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.574882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.574894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.574941 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.575569 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.576078 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.577291 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.577576 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.577638 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.580712 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.580714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.580715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.580721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.583015 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.583603 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.583604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.583604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.583654 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.583836 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.583837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.583837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.583949 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.583993 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.584161 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.584228 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:47:07.584612 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.574196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:41777 (local address 127.3.222.62:41825)
0928 17:47:07.574243 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.574245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.574450 (+ 205us) server_negotiation.cc:407] Connection header received
0928 17:47:07.574712 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.574713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.574725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.574780 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.576671 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.577129 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.578071 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.578363 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.578399 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.581186 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.581187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.581187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.581197 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.582718 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.583090 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.583091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.583092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.583146 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.583335 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.583336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.583337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.583464 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.583525 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.584446 (+ 921us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.584516 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:47:07.584916 11290 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41825
I20260928 17:47:07.585016 11290 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:07.585184 11290 heartbeater.cc:511] Master 127.3.222.62:41825 requested a full tablet report, sending...
I20260928 17:47:07.585352 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.573912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.3.222.3:45195)
0928 17:47:07.574780 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.574789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.574836 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.574954 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.574954 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.575262 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.575483 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.575488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.576158 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.576158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.577215 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.577217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.577698 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.577699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.577826 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.578467 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.578471 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.580622 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.583092 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.583092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.583093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.583533 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.583720 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.583721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.583721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.583772 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.584022 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.584023 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.584061 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.585081 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.585174 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":64,"thread_start_us":72,"threads_started":1}
I20260928 17:47:07.585556 10046 ts_manager.cc:194] Re-registered known tserver with Master: f45792f6552b400b817ec4742a71836a (127.3.222.3:35127)
I20260928 17:47:07.585597 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.574254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:41333 (local address 127.3.222.60:42451)
0928 17:47:07.574293 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.574294 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.574303 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:07.574727 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.574728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.574739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.574786 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.576158 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.576600 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.577488 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.577693 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.577749 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.581142 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.581144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.581144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.581150 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.582560 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.583004 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.583005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.583005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.583061 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.584918 (+ 1857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.584919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.584919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.585060 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.585121 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.585436 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.585501 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:07.585948 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.573912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42451 (local address 127.3.222.3:41333)
0928 17:47:07.574084 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.574122 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.574636 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.574848 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.574852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.575262 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:07.575483 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.575488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.576670 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.576671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.577424 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.577426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.577800 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.577801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.577858 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.578359 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:07.578369 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:07.581058 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:07.582644 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.582651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.582653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.582930 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.583133 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:07.583134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:07.583134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:07.583278 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:07.585149 (+ 1871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:07.585150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:07.585190 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.585246 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.585326 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260928 17:47:07.586020 10046 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:41777
I20260928 17:47:07.586223 11289 heartbeater.cc:347] Connected to a master server at 127.3.222.60:42451
I20260928 17:47:07.586279 11291 heartbeater.cc:347] Connected to a master server at 127.3.222.61:45657
I20260928 17:47:07.586324 11291 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:07.586383 11291 heartbeater.cc:511] Master 127.3.222.61:45657 requested a full tablet report, sending...
I20260928 17:47:07.586278 11289 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:07.586645 10644 ts_manager.cc:194] Registered new tserver with Master: f45792f6552b400b817ec4742a71836a (127.3.222.3:35127)
I20260928 17:47:07.586701 11289 heartbeater.cc:511] Master 127.3.222.60:42451 requested a full tablet report, sending...
I20260928 17:47:07.586953 10836 ts_manager.cc:194] Registered new tserver with Master: f45792f6552b400b817ec4742a71836a (127.3.222.3:35127)
I20260928 17:47:07.587994 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.579271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.3.222.61:45657)
0928 17:47:07.579399 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.579401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.580164 (+ 763us) server_negotiation.cc:407] Connection header received
0928 17:47:07.580259 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.580260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.580272 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.580309 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.580990 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.581684 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.582676 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.582955 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.582990 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.584328 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.584329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.584330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.584335 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.585799 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.586510 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.586511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.586511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.586564 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.587533 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.587534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.587534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.587645 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.587728 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.587828 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.587892 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":48,"threads_started":1}
I20260928 17:47:07.590806 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.579564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.3.222.62:41825)
0928 17:47:07.579796 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.579797 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.579804 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:07.579822 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.579823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.579834 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.580644 (+ 810us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.580682 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.583647 (+ 2965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.584772 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.586915 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.587007 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.588245 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.588246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.588246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.588252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.589774 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.590068 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.590069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.590069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.590142 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.590370 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.590371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.590371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.590467 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.590544 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.590648 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.590721 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260928 17:47:07.592990 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.579851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.3.222.60:42451)
0928 17:47:07.585664 (+ 5813us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.585666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.585673 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:07.585690 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.585690 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.585699 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.585747 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.587352 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.588038 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.588920 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.589190 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.589245 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.590845 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.590846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.590846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.590852 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.592161 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.592474 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.592474 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.592474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.592513 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.592649 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.592650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.592650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.592725 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.592781 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.592887 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.592934 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5800,"thread_start_us":59,"threads_started":1}
I20260928 17:47:07.592139 10046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:47:07.594012 10046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:47:07.600333 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.597154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:46439 (local address 127.0.0.1:38692)
0928 17:47:07.597228 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.597237 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.597279 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.597724 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.597724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.597726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:07.597763 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.597764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.598598 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.598598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.599367 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.599369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.600107 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.600108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.600172 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.600175 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.600227 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.600268 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":36,"threads_started":1}
I20260928 17:47:07.600347 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.597251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.3.222.2:46439)
0928 17:47:07.597398 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.597403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.597413 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:07.597453 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.597455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.597472 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.597735 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:07.597813 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.598535 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.599456 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.600035 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.600096 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.600168 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.600241 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.600276 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260928 17:47:07.601051 11087 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7fa3a45a964b7e99c1590a2beab1fc (DEFAULT_TABLE table=test-table [id=7196d88d6509492ebc558464c3b52a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:07.601342 11087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7fa3a45a964b7e99c1590a2beab1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:07.601696 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.597271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35127 (local address 127.0.0.1:34212)
0928 17:47:07.597598 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.597607 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.597647 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.598223 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.598224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.598226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:07.598258 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.598259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.599457 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.599458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.600655 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.600658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.601480 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.601481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.601540 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.601544 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.601595 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.601633 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":5,"thread_start_us":66,"threads_started":1}
I20260928 17:47:07.601927 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.597382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.3.222.1:36007)
0928 17:47:07.597521 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.597526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.597636 (+ 110us) server_negotiation.cc:407] Connection header received
0928 17:47:07.597676 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.597678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.597692 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.598214 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:07.598399 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.599461 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.600815 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.601478 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.601581 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.601651 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.601722 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.601761 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260928 17:47:07.602188 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.597429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.3.222.3:35127)
0928 17:47:07.597867 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.597872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.597885 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:47:07.597927 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.597930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.597950 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.598212 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:07.598320 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.599374 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.600732 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.601409 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.601478 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.601547 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.602084 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.602121 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":70,"threads_started":1}
I20260928 17:47:07.601811 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.597212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:36007 (local address 127.0.0.1:51790)
0928 17:47:07.597535 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.597543 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.597592 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.598276 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.598277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.598279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:07.598327 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.598329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.599551 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.599551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.600730 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.600731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.601588 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.601589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.601652 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.601656 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.601707 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.601744 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":11,"thread_start_us":63,"threads_started":1}
I20260928 17:47:07.602845 10951 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7fa3a45a964b7e99c1590a2beab1fc (DEFAULT_TABLE table=test-table [id=7196d88d6509492ebc558464c3b52a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:07.602874 11223 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7fa3a45a964b7e99c1590a2beab1fc (DEFAULT_TABLE table=test-table [id=7196d88d6509492ebc558464c3b52a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:07.603116 10951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7fa3a45a964b7e99c1590a2beab1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:07.603191 11223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7fa3a45a964b7e99c1590a2beab1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:07.604632 11317 tablet_bootstrap.cc:492] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: Bootstrap starting.
I20260928 17:47:07.605250 11317 tablet_bootstrap.cc:654] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:07.605566 11317 log.cc:824] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:07.605813 11318 tablet_bootstrap.cc:492] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: Bootstrap starting.
I20260928 17:47:07.606140 11320 tablet_bootstrap.cc:492] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: Bootstrap starting.
I20260928 17:47:07.606492 11318 tablet_bootstrap.cc:654] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:07.606643 11317 tablet_bootstrap.cc:492] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: No bootstrap required, opened a new log
I20260928 17:47:07.606667 11320 tablet_bootstrap.cc:654] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:07.606709 11317 ts_tablet_manager.cc:1397] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:07.606787 11318 log.cc:824] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:07.606926 11320 log.cc:824] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:07.607538 11320 tablet_bootstrap.cc:492] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: No bootstrap required, opened a new log
I20260928 17:47:07.607613 11320 ts_tablet_manager.cc:1397] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:07.607631 11318 tablet_bootstrap.cc:492] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: No bootstrap required, opened a new log
I20260928 17:47:07.607683 11318 ts_tablet_manager.cc:1397] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:07.609372 11320 raft_consensus.cc:348] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:07.609529 11320 raft_consensus.cc:374] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:07.609570 11320 raft_consensus.cc:729] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45792f6552b400b817ec4742a71836a, State: Initialized, Role: FOLLOWER
I20260928 17:47:07.609549 11317 raft_consensus.cc:348] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:07.609671 11320 consensus_queue.cc:260] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:07.609704 11317 raft_consensus.cc:374] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:07.609738 11317 raft_consensus.cc:729] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5c6d1a25304fe79e860d08d35d55df, State: Initialized, Role: FOLLOWER
I20260928 17:47:07.609850 11317 consensus_queue.cc:260] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:07.609913 11320 ts_tablet_manager.cc:1428] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:47:07.610072 11290 heartbeater.cc:503] Master 127.3.222.62:41825 was elected leader, sending a full tablet report...
I20260928 17:47:07.610095 11317 ts_tablet_manager.cc:1428] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 17:47:07.609992 11318 raft_consensus.cc:348] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:07.610131 11318 raft_consensus.cc:374] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:07.610359 11154 heartbeater.cc:503] Master 127.3.222.62:41825 was elected leader, sending a full tablet report...
I20260928 17:47:07.610160 11318 raft_consensus.cc:729] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47845caf59fc4a6f8253c8b6acd59276, State: Initialized, Role: FOLLOWER
I20260928 17:47:07.610810 11318 consensus_queue.cc:260] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:07.611258 11018 heartbeater.cc:503] Master 127.3.222.62:41825 was elected leader, sending a full tablet report...
I20260928 17:47:07.611042 11318 ts_tablet_manager.cc:1428] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:47:07.632270 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.629922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.3.222.60:42451)
0928 17:47:07.629966 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.629967 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.629974 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:07.629991 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.629992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.630009 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.630061 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:07.630164 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.630643 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.631611 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.632004 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.632066 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.632119 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.632192 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.632217 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:47:07.632277 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.629773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42451 (local address 127.0.0.1:53078)
0928 17:47:07.629816 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.629827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.629867 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.630073 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.630074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.630076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:07.630121 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.630123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.630701 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.630702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.631508 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.631511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.632089 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.632090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.632136 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.632140 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.632186 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.632224 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:47:07.632599 10846 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 17:47:07.632951 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf0798ded714fc493705c56ef52d4cf. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.633049 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: This master's current role is: LEARNER
I20260928 17:47:07.674015 11325 raft_consensus.cc:482] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:07.674134 11325 raft_consensus.cc:504] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:07.674454 11325 leader_election.cc:288] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca5c6d1a25304fe79e860d08d35d55df (127.3.222.2:46439), f45792f6552b400b817ec4742a71836a (127.3.222.3:35127)
I20260928 17:47:07.677153 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.674793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:50615 (local address 127.3.222.3:35127)
0928 17:47:07.674837 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.674838 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.674848 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:07.674873 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.674874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.674883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.674921 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:07.675023 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.675552 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.676482 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.676852 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.676903 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.676914 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.677056 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.677089 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:47:07.677160 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.674691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:57451 (local address 127.3.222.2:46439)
0928 17:47:07.674738 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.674740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.674751 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:47:07.674779 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.674780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.674791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.674843 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:07.675062 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.675557 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.676475 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.676865 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.676927 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.676936 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.677067 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.677100 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:47:07.677160 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.674729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35127 (local address 127.3.222.1:50615)
0928 17:47:07.674782 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.674791 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.674831 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.674934 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.674935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.674936 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:07.674973 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.674974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.675618 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.675619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.676389 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.676391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.676920 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.676921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.676990 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.676993 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.677041 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.677080 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":3}
I20260928 17:47:07.677165 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.674618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:46439 (local address 127.3.222.1:57451)
0928 17:47:07.674671 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:07.674683 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:07.674728 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:07.674893 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:07.674894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:07.674895 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:07.674969 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.674971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.675628 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.675630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.676389 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:07.676391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.676938 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.676938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:07.677002 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.677005 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:07.677053 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 17:47:07.677091 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":42}
I20260928 17:47:07.677510 11107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7fa3a45a964b7e99c1590a2beab1fc" candidate_uuid: "47845caf59fc4a6f8253c8b6acd59276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5c6d1a25304fe79e860d08d35d55df" is_pre_election: true
I20260928 17:47:07.677508 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7fa3a45a964b7e99c1590a2beab1fc" candidate_uuid: "47845caf59fc4a6f8253c8b6acd59276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45792f6552b400b817ec4742a71836a" is_pre_election: true
I20260928 17:47:07.677656 11107 raft_consensus.cc:2454] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47845caf59fc4a6f8253c8b6acd59276 in term 0.
I20260928 17:47:07.677657 11243 raft_consensus.cc:2454] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47845caf59fc4a6f8253c8b6acd59276 in term 0.
I20260928 17:47:07.677860 10906 leader_election.cc:302] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47845caf59fc4a6f8253c8b6acd59276, f45792f6552b400b817ec4742a71836a; no voters:
I20260928 17:47:07.677973 11325 raft_consensus.cc:2781] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:47:07.678057 11325 raft_consensus.cc:482] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:47:07.678084 11325 raft_consensus.cc:3037] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:07.678797 11325 raft_consensus.cc:504] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:07.678893 11325 leader_election.cc:288] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [CANDIDATE]: Term 1 election: Requested vote from peers ca5c6d1a25304fe79e860d08d35d55df (127.3.222.2:46439), f45792f6552b400b817ec4742a71836a (127.3.222.3:35127)
I20260928 17:47:07.679023 11107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7fa3a45a964b7e99c1590a2beab1fc" candidate_uuid: "47845caf59fc4a6f8253c8b6acd59276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5c6d1a25304fe79e860d08d35d55df"
I20260928 17:47:07.679081 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7fa3a45a964b7e99c1590a2beab1fc" candidate_uuid: "47845caf59fc4a6f8253c8b6acd59276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45792f6552b400b817ec4742a71836a"
I20260928 17:47:07.679101 11107 raft_consensus.cc:3037] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:07.679158 11243 raft_consensus.cc:3037] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:07.679761 11107 raft_consensus.cc:2454] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47845caf59fc4a6f8253c8b6acd59276 in term 1.
I20260928 17:47:07.679806 11243 raft_consensus.cc:2454] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47845caf59fc4a6f8253c8b6acd59276 in term 1.
I20260928 17:47:07.679903 10906 leader_election.cc:302] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47845caf59fc4a6f8253c8b6acd59276, ca5c6d1a25304fe79e860d08d35d55df; no voters:
I20260928 17:47:07.679976 11325 raft_consensus.cc:2781] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:07.680099 11325 raft_consensus.cc:686] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 1 LEADER]: Becoming Leader. State: Replica: 47845caf59fc4a6f8253c8b6acd59276, State: Running, Role: LEADER
I20260928 17:47:07.680178 11325 consensus_queue.cc:237] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:07.681205 10044 catalog_manager.cc:5495] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47845caf59fc4a6f8253c8b6acd59276 (127.3.222.1). New cstate: current_term: 1 leader_uuid: "47845caf59fc4a6f8253c8b6acd59276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } health_report { overall_health: HEALTHY } } }
I20260928 17:47:07.681758 11307 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: NON_VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: true } } }
I20260928 17:47:07.681845 11307 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:07.682467 11332 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: attempting to promote NON_VOTER 7c61559e058948049e8f4442540799e2 to VOTER
W20260928 17:47:07.682605 11307 consensus_queue.cc:1159] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b560308dae9e4c938a001df5fa9addad" responder_term: 1 status { last_received { term: 1 index: 10 } last_committed_idx: 9 last_received_current_leader { term: 1 index: 10 } }
I20260928 17:47:07.682709 11332 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } }
I20260928 17:47:07.683058 10654 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Refusing update from remote peer fdf0798ded714fc493705c56ef52d4cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 17:47:07.683190 10846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 LEARNER]: Refusing update from remote peer fdf0798ded714fc493705c56ef52d4cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 17:47:07.683256 11307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 17:47:07.683494 11307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 17:47:07.684049 11331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7c61559e058948049e8f4442540799e2 (127.3.222.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:07.684055 10654 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7c61559e058948049e8f4442540799e2 (127.3.222.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:07.684921 11334 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:07.684962 11332 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:07.685000 11334 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:47:07.685011 11332 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:07.684854 11331 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:07.685075 11331 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:07.685241 11331 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:07.685314 11331 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:07.685343 10846 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7c61559e058948049e8f4442540799e2 (127.3.222.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:07.686057 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdf0798ded714fc493705c56ef52d4cf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:07.686143 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:47:07.693218 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.691294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.3.222.1:36007)
0928 17:47:07.691330 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.691332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.691340 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:07.691394 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.691394 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.691410 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.691462 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:07.691570 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.692002 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.692669 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.692834 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.692873 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.692973 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:07.693075 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:07.693129 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.693145 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.693159 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:47:07.694731 11107 raft_consensus.cc:1261] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [term 1 FOLLOWER]: Refusing update from remote peer 47845caf59fc4a6f8253c8b6acd59276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:07.694737 11243 raft_consensus.cc:1261] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [term 1 FOLLOWER]: Refusing update from remote peer 47845caf59fc4a6f8253c8b6acd59276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:07.694998 11329 consensus_queue.cc:1035] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:07.695124 11325 consensus_queue.cc:1035] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:07.698346 11338 mvcc.cc:205] Tried to move back new op lower bound from 7334369803035971584 to 7334369802979610624. Current Snapshot: MvccSnapshot[applied={T|T < 7334369803035971584}]
I20260928 17:47:07.700357 10044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:07.700444 10044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:07.701540 10044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdf0798ded714fc493705c56ef52d4cf: Sending DeleteTablet for 3 replicas of tablet 1f7fa3a45a964b7e99c1590a2beab1fc
I20260928 17:47:07.701812 10951 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7fa3a45a964b7e99c1590a2beab1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:07 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:51790
I20260928 17:47:07.701885 11087 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7fa3a45a964b7e99c1590a2beab1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:07 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:38692
I20260928 17:47:07.702087 11223 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7fa3a45a964b7e99c1590a2beab1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:07 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:34212
I20260928 17:47:07.702386 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 10014
I20260928 17:47:07.702603 11346 tablet_replica.cc:330] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: stopping tablet replica
I20260928 17:47:07.702693 11346 raft_consensus.cc:2229] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:07.702775 11346 raft_consensus.cc:2258] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:07.702901 11347 tablet_replica.cc:330] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: stopping tablet replica
I20260928 17:47:07.702986 11347 raft_consensus.cc:2229] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:07.703082 11347 raft_consensus.cc:2258] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:07.703595 11346 ts_tablet_manager.cc:1905] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:07.705148 11345 tablet_replica.cc:330] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: stopping tablet replica
I20260928 17:47:07.705242 11345 raft_consensus.cc:2229] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:47:07.705353 11345 raft_consensus.cc:2258] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:07.705374 11346 ts_tablet_manager.cc:1918] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:07.705500 11346 log.cc:1196] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/1f7fa3a45a964b7e99c1590a2beab1fc
I20260928 17:47:07.705741 11345 ts_tablet_manager.cc:1905] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:07.705963 11346 ts_tablet_manager.cc:1939] T 1f7fa3a45a964b7e99c1590a2beab1fc P ca5c6d1a25304fe79e860d08d35d55df: Deleting consensus metadata
I20260928 17:47:07.707202 11345 ts_tablet_manager.cc:1918] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:07.707306 11345 log.cc:1196] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/1f7fa3a45a964b7e99c1590a2beab1fc
W20260928 17:47:07.707607 10820 connection.cc:535] server connection from 127.0.0.1:53078 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:47:07.707536 11345 ts_tablet_manager.cc:1939] T 1f7fa3a45a964b7e99c1590a2beab1fc P 47845caf59fc4a6f8253c8b6acd59276: Deleting consensus metadata
W20260928 17:47:07.707839 11042 connection.cc:535] server connection from 127.0.0.1:38692 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 17:47:07.707988 10904 connection.cc:441] server connection from 127.0.0.1:51790 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51790 (request call id 1) could send its response
I20260928 17:47:07.708091 11347 ts_tablet_manager.cc:1905] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:07.710496 11347 ts_tablet_manager.cc:1918] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:07.710562 11347 log.cc:1196] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/1f7fa3a45a964b7e99c1590a2beab1fc
I20260928 17:47:07.710777 11347 ts_tablet_manager.cc:1939] T 1f7fa3a45a964b7e99c1590a2beab1fc P f45792f6552b400b817ec4742a71836a: Deleting consensus metadata
W20260928 17:47:07.711102 11179 connection.cc:441] server connection from 127.0.0.1:34212 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34212 (request call id 1) could send its response
I20260928 17:47:07.716423 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.709094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.3.222.61:45657)
0928 17:47:07.709133 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.709135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.709206 (+ 71us) server_negotiation.cc:407] Connection header received
0928 17:47:07.709236 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.709237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.709248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.709309 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.709939 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.710701 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.711676 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.711986 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.712062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.713897 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.713898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.713898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.713905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.715285 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.715641 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.715643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.715643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.715707 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.715896 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.715896 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.715897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.716018 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.716091 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.716262 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.716334 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:47:07.717207 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:07.708952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.3.222.60:42451)
0928 17:47:07.708991 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:07.708993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:07.710415 (+ 1422us) server_negotiation.cc:407] Connection header received
0928 17:47:07.711153 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:07.711154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:07.711163 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:07.711189 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:07.711830 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.712399 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.713041 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:07.713303 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:07.713370 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:07.714627 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:07.714628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:07.714629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:07.714635 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:07.716119 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.716570 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.716572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.716572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.716628 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:07.716812 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:07.716813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:07.716813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:07.716913 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:07.716977 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:07.717100 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 17:47:07.717152 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:08.698570 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:08.698228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.1:39003)
0928 17:47:08.698450 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:08.698519 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":62,"threads_started":1}
W20260928 17:47:08.698794 11018 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:41825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
I20260928 17:47:08.698989 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:08.698757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.2:47453)
0928 17:47:08.698859 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:08.698927 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":50,"threads_started":1}
W20260928 17:47:08.699172 11154 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:41825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
I20260928 17:47:08.699625 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:08.699280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.3:40743)
0928 17:47:08.699488 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:08.699583 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":43,"threads_started":1}
W20260928 17:47:08.699785 11290 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:41825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
I20260928 17:47:09.202153 11358 raft_consensus.cc:482] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdf0798ded714fc493705c56ef52d4cf)
I20260928 17:47:09.202280 11358 raft_consensus.cc:504] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } }
I20260928 17:47:09.202544 11358 leader_election.cc:288] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdf0798ded714fc493705c56ef52d4cf (127.3.222.62:41825), 7c61559e058948049e8f4442540799e2 (127.3.222.60:42451)
I20260928 17:47:09.202857 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.202658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60706)
0928 17:47:09.202753 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.202809 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
I20260928 17:47:09.202849 11361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdf0798ded714fc493705c56ef52d4cf)
I20260928 17:47:09.202913 11361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } }
W20260928 17:47:09.203024 10629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
I20260928 17:47:09.203334 11361 leader_election.cc:288] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdf0798ded714fc493705c56ef52d4cf (127.3.222.62:41825), b560308dae9e4c938a001df5fa9addad (127.3.222.61:45657)
I20260928 17:47:09.203529 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.203369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60726)
0928 17:47:09.203480 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.203503 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 17:47:09.203606 10629 leader_election.cc:334] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdf0798ded714fc493705c56ef52d4cf (127.3.222.62:41825): Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
I20260928 17:47:09.203713 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.203151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60716)
0928 17:47:09.203321 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.203375 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":49,"threads_started":1}
W20260928 17:47:09.203822 10821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
I20260928 17:47:09.204233 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.204129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60732)
0928 17:47:09.204179 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.204209 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 17:47:09.204334 10821 leader_election.cc:334] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdf0798ded714fc493705c56ef52d4cf (127.3.222.62:41825): Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
I20260928 17:47:09.205687 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.202797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.3.222.60:42451)
0928 17:47:09.202877 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:09.202879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:09.202891 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:47:09.202915 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:09.202916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:09.202927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:09.202992 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:09.203182 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.204021 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.205029 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.205402 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.205459 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.205470 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:09.205592 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 17:47:09.205628 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":35,"threads_started":1}
I20260928 17:47:09.205698 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.202682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42451 (local address 127.0.0.1:53108)
0928 17:47:09.202797 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.202805 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:09.202856 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:09.203045 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:09.203046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:09.203049 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:09.203106 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.203108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.204106 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.204107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.204953 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.204955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.205471 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.205471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.205536 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.205540 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:09.205583 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:47:09.205627 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260928 17:47:09.205916 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b560308dae9e4c938a001df5fa9addad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7c61559e058948049e8f4442540799e2" is_pre_election: true
I20260928 17:47:09.206017 10846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b560308dae9e4c938a001df5fa9addad in term 1.
I20260928 17:47:09.206104 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.203229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45657 (local address 127.0.0.1:59014)
0928 17:47:09.203422 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.203431 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:09.203477 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:09.203660 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:09.203661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:09.203663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:09.203709 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.203711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.204542 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.204543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.205285 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.205288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.205888 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.205889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.205946 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.205950 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:09.206006 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 17:47:09.206042 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":50,"threads_started":1}
I20260928 17:47:09.206106 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.203402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.3.222.61:45657)
0928 17:47:09.203517 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:09.203519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:09.203526 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:09.203546 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:09.203547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:09.203555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:09.203615 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:09.203802 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.204477 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.205359 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.205719 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.205886 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.205896 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:09.206016 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 17:47:09.206048 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260928 17:47:09.206148 10630 leader_election.cc:302] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c61559e058948049e8f4442540799e2, b560308dae9e4c938a001df5fa9addad; no voters: fdf0798ded714fc493705c56ef52d4cf
I20260928 17:47:09.206233 11358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 17:47:09.206269 11358 raft_consensus.cc:482] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdf0798ded714fc493705c56ef52d4cf)
I20260928 17:47:09.206287 11358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:47:09.206332 10654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c61559e058948049e8f4442540799e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b560308dae9e4c938a001df5fa9addad" is_pre_election: true
I20260928 17:47:09.206957 11358 raft_consensus.cc:504] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } }
I20260928 17:47:09.207083 11358 leader_election.cc:288] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [CANDIDATE]: Term 2 election: Requested vote from peers fdf0798ded714fc493705c56ef52d4cf (127.3.222.62:41825), 7c61559e058948049e8f4442540799e2 (127.3.222.60:42451)
I20260928 17:47:09.207089 10654 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c61559e058948049e8f4442540799e2 in current term 2: Already voted for candidate b560308dae9e4c938a001df5fa9addad in this term.
I20260928 17:47:09.207269 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.207178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60736)
0928 17:47:09.207223 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.207249 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 17:47:09.207247 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b560308dae9e4c938a001df5fa9addad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7c61559e058948049e8f4442540799e2"
I20260928 17:47:09.207330 10846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:47:09.207391 10820 leader_election.cc:302] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c61559e058948049e8f4442540799e2; no voters: b560308dae9e4c938a001df5fa9addad, fdf0798ded714fc493705c56ef52d4cf
I20260928 17:47:09.207604 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.207527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60752)
0928 17:47:09.207559 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.207585 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 17:47:09.207664 10629 leader_election.cc:334] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdf0798ded714fc493705c56ef52d4cf (127.3.222.62:41825): Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
I20260928 17:47:09.208011 10846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b560308dae9e4c938a001df5fa9addad in term 2.
I20260928 17:47:09.208092 11361 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:47:09.208189 10630 leader_election.cc:302] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c61559e058948049e8f4442540799e2, b560308dae9e4c938a001df5fa9addad; no voters: fdf0798ded714fc493705c56ef52d4cf
I20260928 17:47:09.208267 11358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:47:09.208402 11358 raft_consensus.cc:686] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [term 2 LEADER]: Becoming Leader. State: Replica: b560308dae9e4c938a001df5fa9addad, State: Running, Role: LEADER
I20260928 17:47:09.208467 11358 consensus_queue.cc:237] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } }
I20260928 17:47:09.208750 11368 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: SysCatalogTable state changed. Reason: New leader b560308dae9e4c938a001df5fa9addad. Latest consensus state: current_term: 2 leader_uuid: "b560308dae9e4c938a001df5fa9addad" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:09.208817 11368 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:09.208942 11370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:09.209314 11370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:09.209379 11370 catalog_manager.cc:1259] Loaded cluster ID: 924fdf473e524903a4fe69f6e2ebc043
I20260928 17:47:09.209412 11370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:47:09.209553 11370 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:09.209619 11370 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Loaded TSK: 0
I20260928 17:47:09.209744 11370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:47:09.243646 10643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:47:09.245301 10643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:47:09.246240 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.246134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60762)
0928 17:47:09.246182 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.246214 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 17:47:09.246234 10846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [term 2 FOLLOWER]: Refusing update from remote peer b560308dae9e4c938a001df5fa9addad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 17:47:09.246481 11368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 17:47:09.246570 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.246487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.0.0.1:60772)
0928 17:47:09.246524 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.246548 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 17:47:09.246690 10629 consensus_peers.cc:487] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad -> Peer fdf0798ded714fc493705c56ef52d4cf (127.3.222.62:41825): Couldn't send request to peer fdf0798ded714fc493705c56ef52d4cf. Status: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 17:47:09.247007 11361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b560308dae9e4c938a001df5fa9addad. Latest consensus state: current_term: 2 leader_uuid: "b560308dae9e4c938a001df5fa9addad" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:09.247128 11361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:47:09.247437 11371 mvcc.cc:205] Tried to move back new op lower bound from 7334369809391546368 to 7334369809238929408. Current Snapshot: MvccSnapshot[applied={T|T < 7334369809391546368}]
I20260928 17:47:09.247516 11358 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b560308dae9e4c938a001df5fa9addad" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:09.247606 11358 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:09.247776 11361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b560308dae9e4c938a001df5fa9addad" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fdf0798ded714fc493705c56ef52d4cf" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41825 } } peers { permanent_uuid: "b560308dae9e4c938a001df5fa9addad" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45657 } attrs { promote: false } } peers { permanent_uuid: "7c61559e058948049e8f4442540799e2" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42451 } attrs { promote: false } } }
I20260928 17:47:09.247864 11361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c61559e058948049e8f4442540799e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:47:09.252154 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.249322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35127 (local address 127.0.0.1:34220)
0928 17:47:09.249446 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.249453 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:09.249484 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:09.249646 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:09.249647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:09.249650 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:09.249698 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.249700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.250582 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.250583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.251421 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.251423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.251975 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.251976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.252036 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.252040 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:09.252069 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 17:47:09.252105 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":60}
I20260928 17:47:09.252380 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.249427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.3.222.3:35127)
0928 17:47:09.249541 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:09.249543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:09.249550 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:09.249571 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:09.249571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:09.249579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:09.249636 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:09.249773 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.250509 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.251500 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.251893 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.252248 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.252275 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:09.252292 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 17:47:09.252332 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260928 17:47:09.252561 11223 tablet_service.cc:1467] Processing CreateTablet for tablet 8935d426171e488f8bdf0fc2748a202f (DEFAULT_TABLE table=test-table [id=bc28b6af38544bc5a242aa348ea55538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:09.252662 11223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8935d426171e488f8bdf0fc2748a202f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:09.252708 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.249242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:46439 (local address 127.0.0.1:38706)
0928 17:47:09.249352 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.249362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:09.249394 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:09.249883 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:09.249884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:09.249885 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:09.249922 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.249923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.250646 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.250647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.251798 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.251799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.252524 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.252525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.252583 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.252587 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:09.252613 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 17:47:09.252647 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":2}
I20260928 17:47:09.252969 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.249323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.3.222.2:46439)
0928 17:47:09.249767 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:09.249769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:09.249775 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:47:09.249799 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:09.249800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:09.249808 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:09.249857 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:09.249975 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.250550 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.251888 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.252436 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.252828 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.252841 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:09.252860 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 17:47:09.252894 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":71,"threads_started":1}
I20260928 17:47:09.253170 11087 tablet_service.cc:1467] Processing CreateTablet for tablet 8935d426171e488f8bdf0fc2748a202f (DEFAULT_TABLE table=test-table [id=bc28b6af38544bc5a242aa348ea55538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:09.253283 11087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8935d426171e488f8bdf0fc2748a202f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:09.253335 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.249535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.3.222.1:36007)
0928 17:47:09.249635 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:09.249637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:09.249672 (+ 35us) server_negotiation.cc:407] Connection header received
0928 17:47:09.249728 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:09.249729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:09.249739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:09.249801 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:09.249903 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.250604 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.251889 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.252918 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.252970 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.252980 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:09.253230 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 17:47:09.253269 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":46,"threads_started":1}
I20260928 17:47:09.253494 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.249333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:36007 (local address 127.0.0.1:51804)
0928 17:47:09.249627 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.249635 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:09.249683 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:09.249802 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:09.249802 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:09.249804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:09.249829 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.249829 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.250673 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.250674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.251796 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.251799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.253116 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.253117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.253159 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.253163 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:09.253210 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 17:47:09.253295 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":27,"thread_start_us":60,"threads_started":1}
I20260928 17:47:09.253891 11381 tablet_bootstrap.cc:492] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: Bootstrap starting.
I20260928 17:47:09.254212 11381 tablet_bootstrap.cc:654] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:09.254483 11382 tablet_bootstrap.cc:492] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df: Bootstrap starting.
I20260928 17:47:09.254781 11381 tablet_bootstrap.cc:492] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: No bootstrap required, opened a new log
I20260928 17:47:09.254807 11382 tablet_bootstrap.cc:654] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:09.254827 11381 ts_tablet_manager.cc:1397] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:09.254868 10951 tablet_service.cc:1467] Processing CreateTablet for tablet 8935d426171e488f8bdf0fc2748a202f (DEFAULT_TABLE table=test-table [id=bc28b6af38544bc5a242aa348ea55538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:09.254920 11381 raft_consensus.cc:348] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:09.254964 10951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8935d426171e488f8bdf0fc2748a202f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:09.254976 11381 raft_consensus.cc:374] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:09.254993 11381 raft_consensus.cc:729] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45792f6552b400b817ec4742a71836a, State: Initialized, Role: FOLLOWER
I20260928 17:47:09.255048 11381 consensus_queue.cc:260] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:09.255190 11381 ts_tablet_manager.cc:1428] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:09.255342 11382 tablet_bootstrap.cc:492] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df: No bootstrap required, opened a new log
I20260928 17:47:09.255384 11382 ts_tablet_manager.cc:1397] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:09.255499 11382 raft_consensus.cc:348] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:09.255553 11382 raft_consensus.cc:374] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:09.255571 11382 raft_consensus.cc:729] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5c6d1a25304fe79e860d08d35d55df, State: Initialized, Role: FOLLOWER
I20260928 17:47:09.255607 11382 consensus_queue.cc:260] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:09.255736 11382 ts_tablet_manager.cc:1428] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:09.256290 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.255934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.2:42015)
0928 17:47:09.256204 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.256238 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":65,"threads_started":1}
I20260928 17:47:09.256291 11155 heartbeater.cc:503] Master 127.3.222.61:45657 was elected leader, sending a full tablet report...
I20260928 17:47:09.256702 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.256474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.2:32789)
0928 17:47:09.256578 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.256677 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260928 17:47:09.256831 11389 tablet_bootstrap.cc:492] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276: Bootstrap starting.
I20260928 17:47:09.256837 11291 heartbeater.cc:503] Master 127.3.222.61:45657 was elected leader, sending a full tablet report...
I20260928 17:47:09.257117 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.256039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.3:58143)
0928 17:47:09.257049 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.257080 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":72,"threads_started":1}
I20260928 17:47:09.257164 11389 tablet_bootstrap.cc:654] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:09.257423 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.257343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.3:36293)
0928 17:47:09.257380 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.257402 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:47:09.257745 11389 tablet_bootstrap.cc:492] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276: No bootstrap required, opened a new log
I20260928 17:47:09.257795 11389 ts_tablet_manager.cc:1397] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:09.257897 11389 raft_consensus.cc:348] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:09.257946 11389 raft_consensus.cc:374] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:09.257957 11389 raft_consensus.cc:729] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47845caf59fc4a6f8253c8b6acd59276, State: Initialized, Role: FOLLOWER
I20260928 17:47:09.257992 11389 consensus_queue.cc:260] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:09.258108 11389 ts_tablet_manager.cc:1428] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:09.258522 11019 heartbeater.cc:503] Master 127.3.222.61:45657 was elected leader, sending a full tablet report...
I20260928 17:47:09.258590 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.258312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.1:45583)
0928 17:47:09.258485 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.258521 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
I20260928 17:47:09.258867 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.258774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.1:34963)
0928 17:47:09.258817 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.258844 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 17:47:09.310056 11020 tablet.cc:2369] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:47:09.325337 11292 tablet.cc:2369] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:47:09.433591 11386 raft_consensus.cc:482] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:09.433692 11386 raft_consensus.cc:504] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:09.434028 11386 leader_election.cc:288] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45792f6552b400b817ec4742a71836a (127.3.222.3:35127), 47845caf59fc4a6f8253c8b6acd59276 (127.3.222.1:36007)
I20260928 17:47:09.436695 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.434212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:37623 (local address 127.3.222.3:35127)
0928 17:47:09.434256 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:09.434258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:09.434367 (+ 109us) server_negotiation.cc:407] Connection header received
0928 17:47:09.434418 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:09.434419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:09.434431 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:09.434476 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:09.434607 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.435066 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.436053 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.436402 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.436459 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.436471 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:09.436599 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 17:47:09.436631 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:47:09.436707 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.434256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:51455 (local address 127.3.222.1:36007)
0928 17:47:09.434313 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:09.434315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:09.434323 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:09.434374 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:09.434375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:09.434388 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:09.434429 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:09.434560 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.435038 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.436044 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.436406 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.436463 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.436474 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:09.436606 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 17:47:09.436644 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:47:09.436713 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.434204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:36007 (local address 127.3.222.2:51455)
0928 17:47:09.434269 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.434278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:09.434324 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:09.434440 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:09.434441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:09.434443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:09.434509 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.434510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.435108 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.435109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.435963 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.435966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.436493 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.436494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.436549 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.436552 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:09.436595 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:47:09.436634 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":25}
I20260928 17:47:09.436715 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.434180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35127 (local address 127.3.222.2:37623)
0928 17:47:09.434290 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.434300 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:09.434360 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:09.434485 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:09.434485 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:09.434487 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:09.434553 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.434554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.435142 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.435143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.435968 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:09.435970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.436471 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.436473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:09.436538 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.436542 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:09.436586 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:47:09.436625 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 17:47:09.436924 10971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8935d426171e488f8bdf0fc2748a202f" candidate_uuid: "ca5c6d1a25304fe79e860d08d35d55df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47845caf59fc4a6f8253c8b6acd59276" is_pre_election: true
I20260928 17:47:09.437052 10971 raft_consensus.cc:2454] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5c6d1a25304fe79e860d08d35d55df in term 0.
I20260928 17:47:09.437130 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8935d426171e488f8bdf0fc2748a202f" candidate_uuid: "ca5c6d1a25304fe79e860d08d35d55df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45792f6552b400b817ec4742a71836a" is_pre_election: true
I20260928 17:47:09.437224 11243 raft_consensus.cc:2454] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5c6d1a25304fe79e860d08d35d55df in term 0.
I20260928 17:47:09.437212 11041 leader_election.cc:302] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47845caf59fc4a6f8253c8b6acd59276, ca5c6d1a25304fe79e860d08d35d55df; no voters:
I20260928 17:47:09.437318 11386 raft_consensus.cc:2781] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:47:09.437355 11386 raft_consensus.cc:482] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:47:09.437372 11386 raft_consensus.cc:3037] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:09.437986 11386 raft_consensus.cc:504] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:09.438094 11386 leader_election.cc:288] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [CANDIDATE]: Term 1 election: Requested vote from peers f45792f6552b400b817ec4742a71836a (127.3.222.3:35127), 47845caf59fc4a6f8253c8b6acd59276 (127.3.222.1:36007)
I20260928 17:47:09.438249 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8935d426171e488f8bdf0fc2748a202f" candidate_uuid: "ca5c6d1a25304fe79e860d08d35d55df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45792f6552b400b817ec4742a71836a"
I20260928 17:47:09.438285 10971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8935d426171e488f8bdf0fc2748a202f" candidate_uuid: "ca5c6d1a25304fe79e860d08d35d55df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47845caf59fc4a6f8253c8b6acd59276"
I20260928 17:47:09.438323 11243 raft_consensus.cc:3037] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:09.438342 10971 raft_consensus.cc:3037] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:09.438997 11243 raft_consensus.cc:2454] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5c6d1a25304fe79e860d08d35d55df in term 1.
I20260928 17:47:09.439082 10971 raft_consensus.cc:2454] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5c6d1a25304fe79e860d08d35d55df in term 1.
I20260928 17:47:09.439175 11042 leader_election.cc:302] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5c6d1a25304fe79e860d08d35d55df, f45792f6552b400b817ec4742a71836a; no voters:
I20260928 17:47:09.439280 11386 raft_consensus.cc:2781] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:09.439421 11386 raft_consensus.cc:686] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [term 1 LEADER]: Becoming Leader. State: Replica: ca5c6d1a25304fe79e860d08d35d55df, State: Running, Role: LEADER
I20260928 17:47:09.439487 11386 consensus_queue.cc:237] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } }
I20260928 17:47:09.439678 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.439575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.2:37071)
0928 17:47:09.439616 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.439651 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 17:47:09.440124 10643 catalog_manager.cc:5495] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5c6d1a25304fe79e860d08d35d55df (127.3.222.2). New cstate: current_term: 1 leader_uuid: "ca5c6d1a25304fe79e860d08d35d55df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5c6d1a25304fe79e860d08d35d55df" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 } health_report { overall_health: UNKNOWN } } }
W20260928 17:47:09.453408 11156 tablet.cc:2369] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:47:09.477207 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.475153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.3.222.2:46439)
0928 17:47:09.475242 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:09.475244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:09.475252 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:09.475284 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:09.475285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:09.475303 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:09.475364 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:09.475490 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.475954 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.476636 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:09.476821 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:09.476867 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:09.476975 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:09.477071 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:09.477118 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:09.477136 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 17:47:09.477152 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 17:47:09.478240 11243 raft_consensus.cc:1261] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [term 1 FOLLOWER]: Refusing update from remote peer ca5c6d1a25304fe79e860d08d35d55df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:09.478286 10971 raft_consensus.cc:1261] T 8935d426171e488f8bdf0fc2748a202f P 47845caf59fc4a6f8253c8b6acd59276 [term 1 FOLLOWER]: Refusing update from remote peer ca5c6d1a25304fe79e860d08d35d55df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:09.478466 11386 consensus_queue.cc:1035] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45792f6552b400b817ec4742a71836a" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:09.478592 11394 consensus_queue.cc:1035] T 8935d426171e488f8bdf0fc2748a202f P ca5c6d1a25304fe79e860d08d35d55df [LEADER]: Connected to new peer: Peer: permanent_uuid: "47845caf59fc4a6f8253c8b6acd59276" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:09.479377 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.479279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.3:35315)
0928 17:47:09.479312 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.479348 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 17:47:09.479564 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.479425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.1:46399)
0928 17:47:09.479505 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.479532 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 17:47:09.480373 11397 mvcc.cc:205] Tried to move back new op lower bound from 7334369810340773888 to 7334369810185592832. Current Snapshot: MvccSnapshot[applied={T|T < 7334369810340773888 or (T in {7334369810340773888})}]
I20260928 17:47:09.480785 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.480700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.1:53723)
0928 17:47:09.480736 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.480759 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 17:47:09.481155 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.481059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.3:42405)
0928 17:47:09.481099 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.481123 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:47:09.481324 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.481235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.2:47113)
0928 17:47:09.481274 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.481297 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:47:09.481499 11399 mvcc.cc:205] Tried to move back new op lower bound from 7334369810340773888 to 7334369810185592832. Current Snapshot: MvccSnapshot[applied={T|T < 7334369810340773888}]
I20260928 17:47:09.481654 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.481571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41825 (local address 127.3.222.2:58247)
0928 17:47:09.481610 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.481633 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 17:47:09.482080 10644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:09.482167 10644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:09.483378 10644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b560308dae9e4c938a001df5fa9addad: Sending DeleteTablet for 3 replicas of tablet 8935d426171e488f8bdf0fc2748a202f
I20260928 17:47:09.483599 10951 tablet_service.cc:1514] Processing DeleteTablet for tablet 8935d426171e488f8bdf0fc2748a202f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:09 UTC) from {username='oryx', principal='oryx/127.3.222.61@KRBTEST.COM'} at 127.0.0.1:51804
I20260928 17:47:09.483718 11223 tablet_service.cc:1514] Processing DeleteTablet for tablet 8935d426171e488f8bdf0fc2748a202f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:09 UTC) from {username='oryx', principal='oryx/127.3.222.61@KRBTEST.COM'} at 127.0.0.1:34220
I20260928 17:47:09.483738 10950 tablet_service.cc:1514] Processing DeleteTablet for tablet 8935d426171e488f8bdf0fc2748a202f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:09 UTC) from {username='oryx', principal='oryx/127.3.222.61@KRBTEST.COM'} at 127.0.0.1:51804
I20260928 17:47:09.483930 11406 tablet_replica.cc:330] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: stopping tablet replica
I20260928 17:47:09.483922 11087 tablet_service.cc:1514] Processing DeleteTablet for tablet 8935d426171e488f8bdf0fc2748a202f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:09 UTC) from {username='oryx', principal='oryx/127.3.222.61@KRBTEST.COM'} at 127.0.0.1:38706
I20260928 17:47:09.483987 11406 raft_consensus.cc:2229] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:09.484027 11406 raft_consensus.cc:2258] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:09.484109 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 10888
I20260928 17:47:09.484728 11406 ts_tablet_manager.cc:1905] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:09.486388 11406 ts_tablet_manager.cc:1918] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:09.486445 11406 log.cc:1196] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/8935d426171e488f8bdf0fc2748a202f
I20260928 17:47:09.486666 11406 ts_tablet_manager.cc:1939] T 8935d426171e488f8bdf0fc2748a202f P f45792f6552b400b817ec4742a71836a: Deleting consensus metadata
I20260928 17:47:09.486938 10630 catalog_manager.cc:4841] TS f45792f6552b400b817ec4742a71836a (127.3.222.3:35127): tablet 8935d426171e488f8bdf0fc2748a202f (table test-table [id=bc28b6af38544bc5a242aa348ea55538]) successfully deleted
I20260928 17:47:09.489832 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.489717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:36007 (local address 127.0.0.1:51806)
0928 17:47:09.489768 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.489801 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:47:09.489945 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 11024
W20260928 17:47:09.489966 10629 catalog_manager.cc:4584] TS 47845caf59fc4a6f8253c8b6acd59276 (127.3.222.1:36007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8935d426171e488f8bdf0fc2748a202f: Network error: Client connection negotiation failed: client connection to 127.3.222.1:36007: connect: Connection refused (error 111)
I20260928 17:47:09.494334 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 11160
I20260928 17:47:09.494364 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:09.494254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:46439 (local address 127.0.0.1:38722)
0928 17:47:09.494307 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:09.494335 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.2:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:47:09.498034 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 10510
I20260928 17:47:09.501718 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 10698
2026-09-28T17:47:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4563 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:47:09 dist-test-slave-w3bm krb5kdc[11411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:47:09 dist-test-slave-w3bm krb5kdc[11411](info): set up 2 sockets
Sep 28 17:47:09 dist-test-slave-w3bm krb5kdc[11411](info): commencing operation
krb5kdc: starting...
W20260928 17:47:10.451365 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:47:10 dist-test-slave-w3bm krb5kdc[11411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:47:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.62.keytab.
Entry for principal oryx/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.62.keytab.
I20260928 17:47:10.561394 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:40167
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.62.keytab
--principal=oryx/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:10.637158 11424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:10.637343 11424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:10.637372 11424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:10.638466 11424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:10.638512 11424 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 17:47:10.638528 11424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:10.638545 11424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:10.638561 11424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:10.638581 11424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:10.640040 11424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:40167
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.62.keytab
--principal=oryx/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:10.640260 11424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:10.640443 11424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:47:10.644965 11424 server_base.cc:1034] running on GCE node
W20260928 17:47:10.645037 11429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:10.644937 11430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:10.644928 11432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:10.645393 11424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:10.645644 11424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:10.646785 11424 hybrid_clock.cc:648] HybridClock initialized: now 1790617630646767 us; error 31 us; skew 500 ppm
Sep 28 17:47:10 dist-test-slave-w3bm krb5kdc[11411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617630, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:10.649143 11424 init.cc:377] Logged in from keytab as oryx/127.3.222.62@KRBTEST.COM (short username oryx)
I20260928 17:47:10.649551 11424 webserver.cc:460] Webserver started at http://127.3.222.62:37473/ using document root <none> and password file <none>
I20260928 17:47:10.649731 11424 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:10.649768 11424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:10.649854 11424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:10.650805 11424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "f8139ccf094a4a89a41d2c990b02ad86"
format_stamp: "Formatted at 2026-09-28 17:47:10 on dist-test-slave-w3bm"
server_key: "3ee34b257dd8b7a6064c21420fa2b149"
server_key_iv: "7b2a05cdafbcd58b94b8d31b7e42ec8b"
server_key_version: "encryptionkey@0"
I20260928 17:47:10.651089 11424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "f8139ccf094a4a89a41d2c990b02ad86"
format_stamp: "Formatted at 2026-09-28 17:47:10 on dist-test-slave-w3bm"
server_key: "3ee34b257dd8b7a6064c21420fa2b149"
server_key_iv: "7b2a05cdafbcd58b94b8d31b7e42ec8b"
server_key_version: "encryptionkey@0"
I20260928 17:47:10.652324 11424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:47:10.652997 11439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:10.653144 11424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:10.653221 11424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "f8139ccf094a4a89a41d2c990b02ad86"
format_stamp: "Formatted at 2026-09-28 17:47:10 on dist-test-slave-w3bm"
server_key: "3ee34b257dd8b7a6064c21420fa2b149"
server_key_iv: "7b2a05cdafbcd58b94b8d31b7e42ec8b"
server_key_version: "encryptionkey@0"
I20260928 17:47:10.653270 11424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:10.663184 11424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:10.664903 11424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:10.665012 11424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:10.669257 11424 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:40167
I20260928 17:47:10.669287 11501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:40167 every 8 connection(s)
I20260928 17:47:10.669576 11424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:10.670128 11502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:10.672217 11502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86: Bootstrap starting.
I20260928 17:47:10.672717 11502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:10.672932 11502 log.cc:824] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:10.673494 11502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86: No bootstrap required, opened a new log
I20260928 17:47:10.675400 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 11424
I20260928 17:47:10.675499 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:10.675411 11502 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8139ccf094a4a89a41d2c990b02ad86" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40167 } }
I20260928 17:47:10.675575 11502 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:10.675616 11502 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8139ccf094a4a89a41d2c990b02ad86, State: Initialized, Role: FOLLOWER
I20260928 17:47:10.675680 3960 external_mini_cluster.cc:1428] Setting key 14c9610f57f29d8c2c660b6825889b63
I20260928 17:47:10.675752 11502 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8139ccf094a4a89a41d2c990b02ad86" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40167 } }
I20260928 17:47:10.675822 11502 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:10.675858 11502 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:10.675896 11502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:10.676476 11502 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8139ccf094a4a89a41d2c990b02ad86" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40167 } }
I20260928 17:47:10.676580 11502 leader_election.cc:302] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8139ccf094a4a89a41d2c990b02ad86; no voters:
I20260928 17:47:10.676723 11502 leader_election.cc:288] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:10.676760 11507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:10.676875 11507 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [term 1 LEADER]: Becoming Leader. State: Replica: f8139ccf094a4a89a41d2c990b02ad86, State: Running, Role: LEADER
I20260928 17:47:10.676956 11502 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:10.676959 11507 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8139ccf094a4a89a41d2c990b02ad86" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40167 } }
I20260928 17:47:10.677282 11508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8139ccf094a4a89a41d2c990b02ad86. Latest consensus state: current_term: 1 leader_uuid: "f8139ccf094a4a89a41d2c990b02ad86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8139ccf094a4a89a41d2c990b02ad86" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40167 } } }
I20260928 17:47:10.677372 11508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:10.677573 11513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:10.678066 11513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:10.678840 11509 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8139ccf094a4a89a41d2c990b02ad86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8139ccf094a4a89a41d2c990b02ad86" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40167 } } }
I20260928 17:47:10.678910 11509 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:10.680442 11513 catalog_manager.cc:1347] Generated new cluster ID: 649eb05ebb5e48eb8a79303414431991
I20260928 17:47:10.680491 11513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:47:10 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:10.686148 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:10.676084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.3.222.62:40167)
0928 17:47:10.676238 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:10.676245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:10.676270 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:47:10.676360 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:10.676365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:10.676402 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:10.676503 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:10.677251 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.678485 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.679468 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.680027 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.680122 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:10.682719 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:10.682723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:10.682723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:10.682735 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:10.684559 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:10.684900 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:10.684904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:10.684904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:10.685019 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:10.685183 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:10.685184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:10.685184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:10.685476 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:10.685640 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:10.685705 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 17:47:10.685929 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260928 17:47:10.691633 11513 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:10.692310 11513 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:10.698977 11513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86: Generated new TSK 0
I20260928 17:47:10.699138 11513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab.
Entry for principal oryx/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab.
I20260928 17:47:10.730790 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:40167
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab
--principal=oryx/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:10.808437 11528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:10.808645 11528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:10.808668 11528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:10.809759 11528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:10.809793 11528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:10.809811 11528 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:47:10.809826 11528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:47:10.811250 11528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab
--principal=oryx/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.222.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:10.811435 11528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:10.811601 11528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:10.816208 11533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:10.816355 11528 server_base.cc:1034] running on GCE node
W20260928 17:47:10.816195 11534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:10.816190 11536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:10.816643 11528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:10.816864 11528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:10.818008 11528 hybrid_clock.cc:648] HybridClock initialized: now 1790617630817994 us; error 26 us; skew 500 ppm
Sep 28 17:47:10 dist-test-slave-w3bm krb5kdc[11411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617630, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:10.820400 11528 init.cc:377] Logged in from keytab as oryx/127.3.222.1@KRBTEST.COM (short username oryx)
I20260928 17:47:10.820864 11528 webserver.cc:460] Webserver started at http://127.3.222.1:46823/ using document root <none> and password file <none>
I20260928 17:47:10.821089 11528 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:10.821144 11528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:10.821266 11528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:10.822309 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "7fd1f5b9293f48caba6fc5c6797d0dec"
format_stamp: "Formatted at 2026-09-28 17:47:10 on dist-test-slave-w3bm"
server_key: "df816eb2ee9b02179e72c09da31490d7"
server_key_iv: "316f78263f950669469d180ac6d519b3"
server_key_version: "encryptionkey@0"
I20260928 17:47:10.822598 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "7fd1f5b9293f48caba6fc5c6797d0dec"
format_stamp: "Formatted at 2026-09-28 17:47:10 on dist-test-slave-w3bm"
server_key: "df816eb2ee9b02179e72c09da31490d7"
server_key_iv: "316f78263f950669469d180ac6d519b3"
server_key_version: "encryptionkey@0"
I20260928 17:47:10.823818 11528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 17:47:10.824430 11543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:10.824580 11528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:10.824662 11528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "7fd1f5b9293f48caba6fc5c6797d0dec"
format_stamp: "Formatted at 2026-09-28 17:47:10 on dist-test-slave-w3bm"
server_key: "df816eb2ee9b02179e72c09da31490d7"
server_key_iv: "316f78263f950669469d180ac6d519b3"
server_key_version: "encryptionkey@0"
I20260928 17:47:10.824717 11528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:10.833673 11528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:10.835366 11528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:10.835461 11528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:10.835851 11528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:10.835889 11528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:10.835908 11528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:10.835919 11528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:10.842082 11528 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:44627
I20260928 17:47:10.842231 11658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:44627 every 8 connection(s)
I20260928 17:47:10.842418 11528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:47:10 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617630, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:10.844838 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 11528
I20260928 17:47:10.844942 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:47:10.845127 3960 external_mini_cluster.cc:1428] Setting key f5ab4498c4b1283db458eab7893ebafd
I20260928 17:47:10.845203 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:10.836280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:51265 (local address 127.3.222.62:40167)
0928 17:47:10.836340 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:10.836343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:10.836550 (+ 207us) server_negotiation.cc:407] Connection header received
0928 17:47:10.837057 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:10.837059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:10.837075 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:10.837126 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:10.837987 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.838743 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.840231 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.840499 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.840553 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:10.842988 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:10.842989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:10.842990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:10.842996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:10.844052 (+ 1056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:10.844381 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:10.844382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:10.844382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:10.844423 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:10.844604 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:10.844605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:10.844605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:10.844697 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:10.844742 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:10.845057 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 17:47:10.845126 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:47:10.845397 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:10.836249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40167 (local address 127.3.222.1:51265)
0928 17:47:10.836410 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:10.836448 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:10.836950 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:10.837235 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:10.837241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:10.837574 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:10.837848 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:10.837855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.838830 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.838831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:10.840147 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:10.840151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.840581 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.840582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:10.840723 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:10.841223 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:10.841232 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:10.842908 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:10.844121 (+ 1213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:10.844127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:10.844128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:10.844326 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:10.844468 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:10.844468 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:10.844469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:10.844550 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:10.844760 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:10.844767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:10.844922 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 17:47:10.845040 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 17:47:10.845195 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260928 17:47:10.845845 11659 heartbeater.cc:347] Connected to a master server at 127.3.222.62:40167
I20260928 17:47:10.845925 11659 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:10.846098 11659 heartbeater.cc:511] Master 127.3.222.62:40167 requested a full tablet report, sending...
I20260928 17:47:10.846601 11455 ts_manager.cc:194] Registered new tserver with Master: 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627)
I20260928 17:47:10.847378 11455 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:51265
WARNING: no policy specified for oryx/127.3.222.2@KRBTEST.COM; defaulting to no policy
I20260928 17:47:10.852478 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:10.847537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:45845 (local address 127.3.222.62:40167)
0928 17:47:10.847584 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:10.847586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:10.847593 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:10.847613 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:10.847614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:10.847623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:10.847668 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:10.848038 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.848572 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.849290 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.849495 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.849549 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:10.850316 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:10.850318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:10.850318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:10.850324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:10.851539 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:10.851814 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:10.851815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:10.851815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:10.851866 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:10.852023 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:10.852024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:10.852024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:10.852122 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:10.852174 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:10.852313 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 17:47:10.852382 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 17:47:10.852484 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:10.846785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40167 (local address 127.3.222.1:45845)
0928 17:47:10.846902 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:10.846913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:10.846961 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:10.847677 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:10.847678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:10.847859 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:10.847924 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:10.847927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.848659 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.848660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:10.849223 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:10.849226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.849552 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.849553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:10.849614 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:10.849986 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:10.849990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:10.850243 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:10.851606 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:10.851607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:10.851608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:10.851759 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:10.851915 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:10.851916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:10.851916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:10.851977 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:10.852171 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:10.852172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:10.852228 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 17:47:10.852286 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 17:47:10.852382 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab.
Entry for principal oryx/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab.
I20260928 17:47:10.870494 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:40167
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab
--principal=oryx/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:10.947111 11668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:10.947312 11668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:10.947335 11668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:10.948454 11668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:10.948494 11668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:10.948513 11668 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:47:10.948544 11668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:47:10.949991 11668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab
--principal=oryx/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.222.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:10.950192 11668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:10.950377 11668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:10.955029 11674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:10.955061 11676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:10.955039 11673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:10.955065 11668 server_base.cc:1034] running on GCE node
I20260928 17:47:10.955406 11668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:10.955644 11668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:10.956832 11668 hybrid_clock.cc:648] HybridClock initialized: now 1790617630956821 us; error 33 us; skew 500 ppm
Sep 28 17:47:10 dist-test-slave-w3bm krb5kdc[11411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617630, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:10.959254 11668 init.cc:377] Logged in from keytab as oryx/127.3.222.2@KRBTEST.COM (short username oryx)
I20260928 17:47:10.959663 11668 webserver.cc:460] Webserver started at http://127.3.222.2:46091/ using document root <none> and password file <none>
I20260928 17:47:10.959849 11668 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:10.959893 11668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:10.960011 11668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:10.960965 11668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "c37e322f70254082820f61a9a68a6316"
format_stamp: "Formatted at 2026-09-28 17:47:10 on dist-test-slave-w3bm"
server_key: "4e268984785e00981831432ad674d155"
server_key_iv: "ae21d61349e66a8264281e7340428a24"
server_key_version: "encryptionkey@0"
I20260928 17:47:10.961301 11668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "c37e322f70254082820f61a9a68a6316"
format_stamp: "Formatted at 2026-09-28 17:47:10 on dist-test-slave-w3bm"
server_key: "4e268984785e00981831432ad674d155"
server_key_iv: "ae21d61349e66a8264281e7340428a24"
server_key_version: "encryptionkey@0"
I20260928 17:47:10.962461 11668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:47:10.963110 11683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:10.963261 11668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:10.963379 11668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "c37e322f70254082820f61a9a68a6316"
format_stamp: "Formatted at 2026-09-28 17:47:10 on dist-test-slave-w3bm"
server_key: "4e268984785e00981831432ad674d155"
server_key_iv: "ae21d61349e66a8264281e7340428a24"
server_key_version: "encryptionkey@0"
I20260928 17:47:10.963447 11668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:10.989233 11668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:10.990929 11668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:10.991070 11668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:10.991580 11668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:10.991621 11668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:10.991650 11668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:10.991667 11668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:10.998198 11668 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:35833
I20260928 17:47:10.998332 11798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:35833 every 8 connection(s)
Sep 28 17:47:10 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617630, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:10.998584 11668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:47:11.001289 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:10.992016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:44205 (local address 127.3.222.62:40167)
0928 17:47:10.992077 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:10.992080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:10.992292 (+ 212us) server_negotiation.cc:407] Connection header received
0928 17:47:10.993000 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:10.993001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:10.993011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:10.993053 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:10.993869 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.994640 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.996151 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.996427 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.996487 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:10.998967 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:10.998969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:10.998970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:10.998977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:11.000216 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.000550 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.000551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.000551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.000592 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.000837 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.000838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.000838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.000926 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:11.000981 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.001177 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.001230 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:47:11.001466 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:10.991991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40167 (local address 127.3.222.2:44205)
0928 17:47:10.992155 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:10.992203 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:10.992893 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:10.993134 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:10.993139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:10.993488 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:10.993775 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:10.993791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.994756 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.994757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:10.996073 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:10.996077 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:10.996520 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:10.996521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:10.996669 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:10.997206 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:10.997216 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:10.998855 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:11.000284 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.000290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.000292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.000492 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.000663 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.000664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.000664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.000768 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.000981 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:11.000984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:11.001091 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.001163 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.001278 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260928 17:47:11.001920 11799 heartbeater.cc:347] Connected to a master server at 127.3.222.62:40167
I20260928 17:47:11.001994 11799 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:11.002144 11799 heartbeater.cc:511] Master 127.3.222.62:40167 requested a full tablet report, sending...
I20260928 17:47:11.002480 11455 ts_manager.cc:194] Registered new tserver with Master: c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833)
I20260928 17:47:11.002863 11455 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:44205
I20260928 17:47:11.004745 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 11668
I20260928 17:47:11.004850 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:47:11.005021 3960 external_mini_cluster.cc:1428] Setting key 640ca3ae52742ab2321b6900fc5efb7f
I20260928 17:47:11.007690 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.002858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:59623 (local address 127.3.222.62:40167)
0928 17:47:11.002893 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.002894 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.002952 (+ 58us) server_negotiation.cc:407] Connection header received
0928 17:47:11.003025 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.003026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.003035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.003080 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:11.003395 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.003920 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.004523 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.004689 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.004738 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.005585 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:11.005586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:11.005587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:11.005593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:11.006723 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.006996 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.006997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.006997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.007046 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.007203 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.007218 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.007218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.007350 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:11.007415 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.007542 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.007604 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:47:11.007702 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.002785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40167 (local address 127.3.222.2:59623)
0928 17:47:11.002890 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.002900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.002976 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.003084 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.003085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.003283 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:11.003336 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.003338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.003995 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.003995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.004472 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.004473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.004747 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.004749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.004815 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.005166 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:11.005170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:11.005505 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:11.006792 (+ 1287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.006793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.006793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.006940 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.007100 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.007101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.007101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.007154 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.007405 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:11.007406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:11.007455 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.007515 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.007605 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":52,"threads_started":1}
WARNING: no policy specified for oryx/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab.
Entry for principal oryx/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab.
I20260928 17:47:11.029637 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:40167
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab
--principal=oryx/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:11.108902 11808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:11.109107 11808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:11.109138 11808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:11.110277 11808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:11.110314 11808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:11.110342 11808 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:47:11.110364 11808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:11.111838 11808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab
--principal=oryx/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.222.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:11.112067 11808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:11.112272 11808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:11.116973 11814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:11.117024 11813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:11.117034 11808 server_base.cc:1034] running on GCE node
W20260928 17:47:11.116990 11816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:11.117347 11808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:11.117571 11808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:11.118710 11808 hybrid_clock.cc:648] HybridClock initialized: now 1790617631118694 us; error 34 us; skew 500 ppm
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617631, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:11.121063 11808 init.cc:377] Logged in from keytab as oryx/127.3.222.3@KRBTEST.COM (short username oryx)
I20260928 17:47:11.121457 11808 webserver.cc:460] Webserver started at http://127.3.222.3:43069/ using document root <none> and password file <none>
I20260928 17:47:11.121626 11808 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:11.121662 11808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:11.121747 11808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:11.122673 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "02f4c0609bfb4148b9d437773f2959eb"
format_stamp: "Formatted at 2026-09-28 17:47:11 on dist-test-slave-w3bm"
server_key: "a84f0247ac7f024413f1ab50d6c9a682"
server_key_iv: "3f7e6fab8a809c2c9d7e3aa8b9a10933"
server_key_version: "encryptionkey@0"
I20260928 17:47:11.122944 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "02f4c0609bfb4148b9d437773f2959eb"
format_stamp: "Formatted at 2026-09-28 17:47:11 on dist-test-slave-w3bm"
server_key: "a84f0247ac7f024413f1ab50d6c9a682"
server_key_iv: "3f7e6fab8a809c2c9d7e3aa8b9a10933"
server_key_version: "encryptionkey@0"
I20260928 17:47:11.124130 11808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:47:11.124749 11823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.124909 11808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:11.124986 11808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "02f4c0609bfb4148b9d437773f2959eb"
format_stamp: "Formatted at 2026-09-28 17:47:11 on dist-test-slave-w3bm"
server_key: "a84f0247ac7f024413f1ab50d6c9a682"
server_key_iv: "3f7e6fab8a809c2c9d7e3aa8b9a10933"
server_key_version: "encryptionkey@0"
I20260928 17:47:11.125041 11808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:11.152153 11808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:11.154174 11808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:11.154300 11808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:11.154778 11808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:11.154814 11808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.154842 11808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:11.154863 11808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.161342 11808 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:35919
I20260928 17:47:11.161412 11938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:35919 every 8 connection(s)
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617631, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:11.161695 11808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:47:11.163981 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 11808
I20260928 17:47:11.164072 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:47:11.164265 3960 external_mini_cluster.cc:1428] Setting key 8265286d8655286e39db817afce38ca8
I20260928 17:47:11.164317 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.155407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:48625 (local address 127.3.222.62:40167)
0928 17:47:11.155447 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.155448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.155457 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:11.156100 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.156101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.156127 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.156176 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:11.157025 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.157768 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.159303 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.159578 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.159632 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.161901 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:11.161904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:11.161904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:11.161912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:11.162998 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.163395 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.163397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.163397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.163456 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.163669 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.163670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.163671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.163777 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:11.163831 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.164139 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.164232 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:11.164390 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.155147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40167 (local address 127.3.222.3:48625)
0928 17:47:11.155310 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.155379 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.156001 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.156282 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.156286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.156623 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:11.156917 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.156924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.157846 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.157848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.159175 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.159178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.159639 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.159640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.159779 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.160248 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:11.160257 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:11.161808 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:11.163066 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.163072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.163074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.163326 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.163506 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.163506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.163507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.163609 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.163839 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:11.163845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:11.163979 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.164064 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.164213 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260928 17:47:11.164726 11939 heartbeater.cc:347] Connected to a master server at 127.3.222.62:40167
I20260928 17:47:11.164808 11939 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:11.164992 11939 heartbeater.cc:511] Master 127.3.222.62:40167 requested a full tablet report, sending...
I20260928 17:47:11.165305 11455 ts_manager.cc:194] Registered new tserver with Master: 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919)
I20260928 17:47:11.165783 11455 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:48625
I20260928 17:47:11.170554 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.165695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:50453 (local address 127.3.222.62:40167)
0928 17:47:11.165730 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.165731 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.165831 (+ 100us) server_negotiation.cc:407] Connection header received
0928 17:47:11.165923 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.165924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.165934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.165982 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:11.166286 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.166842 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.167531 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.167702 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.167745 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.168526 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:11.168527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:11.168527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:11.168531 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:11.169597 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.169890 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.169891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.169891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.169939 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.170153 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.170153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.170153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.170233 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:11.170277 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.170447 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.170498 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:11.170567 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.165647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40167 (local address 127.3.222.3:50453)
0928 17:47:11.165759 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.165769 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.165865 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.165992 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.165993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.166152 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:11.166223 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.166224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.166931 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.166931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.167471 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.167472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.167781 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.167782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.167835 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.168221 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:11.168225 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:11.168470 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:11.169666 (+ 1196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.169667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.169668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.169835 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.170027 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.170028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.170028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.170098 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.170301 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:11.170301 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:11.170342 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.170426 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.170499 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":5,"thread_start_us":58,"threads_started":1}
I20260928 17:47:11.176026 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 17:47:11.195128 11506 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:11.190403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.3.222.62:40167)
0928 17:47:11.190465 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.190467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.190475 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:11.190499 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.190500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.190509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.190558 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:11.191032 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.191583 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.192265 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.192438 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.192480 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.195059 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40284: BlockingRecv error: recv got EOF from 127.0.0.1:40284 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:11.201375 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.195777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.3.222.62:40167)
0928 17:47:11.195821 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.195823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.195908 (+ 85us) server_negotiation.cc:407] Connection header received
0928 17:47:11.195945 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.195946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.195953 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.195993 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:11.196349 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.196788 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.197379 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.197541 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.197580 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.199484 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:11.199485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:11.199485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:11.199491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:11.200631 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.200903 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.200903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.200904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.200945 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.201059 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.201060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.201060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.201140 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:11.201199 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.201264 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.201314 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:47:11.202759 11455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:47:11.204293 11455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:47:11.210059 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.206622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.3.222.1:44627)
0928 17:47:11.206818 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.206822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.206830 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:11.206875 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.206878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.206892 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.206957 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:11.207051 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.207865 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.208996 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.209701 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.209754 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.209829 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.209930 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.209968 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":65,"threads_started":1}
I20260928 17:47:11.210058 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.206295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:35833 (local address 127.0.0.1:43236)
0928 17:47:11.206396 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.206434 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.206490 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.206811 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.206814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.206816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:11.206872 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.206873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.208233 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.208234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.209130 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.209132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.209808 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.209809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.209880 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.209884 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.209909 (+ 25us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.209974 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":55,"threads_started":1}
I20260928 17:47:11.210062 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.206296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44627 (local address 127.0.0.1:52546)
0928 17:47:11.206737 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.206747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.206788 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.206962 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.206962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.206964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:11.207004 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.207005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.207928 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.207929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.208906 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.208908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.209764 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.209765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.209844 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.209858 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.209918 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.209981 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":21,"thread_start_us":99,"threads_started":1}
I20260928 17:47:11.210465 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.206402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.3.222.2:35833)
0928 17:47:11.206609 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.206614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.206625 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:47:11.206663 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.206666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.206681 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.206947 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:11.207012 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.208123 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.209199 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.209708 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.210240 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.210318 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.210341 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.210378 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
I20260928 17:47:11.210580 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.206502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35919 (local address 127.0.0.1:35322)
0928 17:47:11.206768 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.206777 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.206854 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.207320 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.207321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.207323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:11.207373 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.207375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.208496 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.208497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.209515 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.209518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.210361 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.210363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.210425 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.210429 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.210486 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.210523 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":61,"threads_started":1}
I20260928 17:47:11.210784 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.206579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.3.222.3:35919)
0928 17:47:11.206745 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.206750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.206827 (+ 77us) server_negotiation.cc:407] Connection header received
0928 17:47:11.207204 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.207222 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.207239 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.207323 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:11.207431 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.208423 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.209597 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.210281 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.210343 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.210400 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.210502 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.210698 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260928 17:47:11.211077 11593 tablet_service.cc:1467] Processing CreateTablet for tablet 06ec2ac672134f8eb1a922fd38095e3d (DEFAULT_TABLE table=test-table [id=aecb25a4ce3348de94f937c96fb4e658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.211483 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ec2ac672134f8eb1a922fd38095e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.211359 11873 tablet_service.cc:1467] Processing CreateTablet for tablet 06ec2ac672134f8eb1a922fd38095e3d (DEFAULT_TABLE table=test-table [id=aecb25a4ce3348de94f937c96fb4e658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.211683 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ec2ac672134f8eb1a922fd38095e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.214952 11967 tablet_bootstrap.cc:492] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: Bootstrap starting.
I20260928 17:47:11.215070 11968 tablet_bootstrap.cc:492] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: Bootstrap starting.
I20260928 17:47:11.215507 11967 tablet_bootstrap.cc:654] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.215514 11732 tablet_service.cc:1467] Processing CreateTablet for tablet 06ec2ac672134f8eb1a922fd38095e3d (DEFAULT_TABLE table=test-table [id=aecb25a4ce3348de94f937c96fb4e658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.215651 11968 tablet_bootstrap.cc:654] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.215793 11967 log.cc:824] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:11.215831 11732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ec2ac672134f8eb1a922fd38095e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.215937 11968 log.cc:824] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:11.216477 11967 tablet_bootstrap.cc:492] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: No bootstrap required, opened a new log
I20260928 17:47:11.216543 11967 ts_tablet_manager.cc:1397] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:11.216591 11968 tablet_bootstrap.cc:492] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: No bootstrap required, opened a new log
I20260928 17:47:11.216653 11968 ts_tablet_manager.cc:1397] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:11.218101 11968 raft_consensus.cc:348] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.218225 11968 raft_consensus.cc:374] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.218255 11968 raft_consensus.cc:729] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd1f5b9293f48caba6fc5c6797d0dec, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.218367 11968 consensus_queue.cc:260] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.218390 11967 raft_consensus.cc:348] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.218540 11967 raft_consensus.cc:374] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.218577 11967 raft_consensus.cc:729] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f4c0609bfb4148b9d437773f2959eb, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.218603 11968 ts_tablet_manager.cc:1428] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:47:11.218675 11967 consensus_queue.cc:260] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.218711 11659 heartbeater.cc:503] Master 127.3.222.62:40167 was elected leader, sending a full tablet report...
I20260928 17:47:11.218927 11967 ts_tablet_manager.cc:1428] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:47:11.218945 11939 heartbeater.cc:503] Master 127.3.222.62:40167 was elected leader, sending a full tablet report...
I20260928 17:47:11.219794 11973 tablet_bootstrap.cc:492] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: Bootstrap starting.
I20260928 17:47:11.220275 11973 tablet_bootstrap.cc:654] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.220521 11973 log.cc:824] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:11.221064 11973 tablet_bootstrap.cc:492] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: No bootstrap required, opened a new log
I20260928 17:47:11.221127 11973 ts_tablet_manager.cc:1397] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:11.222571 11973 raft_consensus.cc:348] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.222684 11973 raft_consensus.cc:374] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.222703 11973 raft_consensus.cc:729] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37e322f70254082820f61a9a68a6316, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.222806 11973 consensus_queue.cc:260] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.223006 11973 ts_tablet_manager.cc:1428] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:47:11.223047 11799 heartbeater.cc:503] Master 127.3.222.62:40167 was elected leader, sending a full tablet report...
W20260928 17:47:11.249066 11800 tablet.cc:2369] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:47:11.343184 11660 tablet.cc:2369] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:47:11.346640 11972 raft_consensus.cc:482] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:11.346741 11972 raft_consensus.cc:504] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.347034 11972 leader_election.cc:288] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627), c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833)
I20260928 17:47:11.349789 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.347257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:45199 (local address 127.3.222.1:44627)
0928 17:47:11.347344 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.347346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.347355 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:11.347384 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.347385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.347397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.347443 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:11.347565 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.348031 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.349082 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.349462 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.349523 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.349535 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.349691 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.349727 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 17:47:11.349798 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.347201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44627 (local address 127.3.222.3:45199)
0928 17:47:11.347275 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.347285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.347330 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.347445 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.347445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.347446 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:11.347511 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.347513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.348095 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.348096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.349005 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.349008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.349545 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.349546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.349634 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.349637 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.349680 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.349718 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":21}
I20260928 17:47:11.349851 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.347294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:55267 (local address 127.3.222.2:35833)
0928 17:47:11.347341 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.347343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.347445 (+ 102us) server_negotiation.cc:407] Connection header received
0928 17:47:11.347507 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.347509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.347521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.347560 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:11.347705 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.348141 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.349216 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.349552 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.349610 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.349619 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.349752 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.349788 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:47:11.349891 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.347237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:35833 (local address 127.3.222.3:55267)
0928 17:47:11.347383 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.347392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.347446 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.347568 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.347569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.347570 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:11.347634 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.347635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.348211 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.348212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.349113 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.349116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.349619 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.349620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.349697 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.349700 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.349742 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.349779 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260928 17:47:11.350148 11613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ec2ac672134f8eb1a922fd38095e3d" candidate_uuid: "02f4c0609bfb4148b9d437773f2959eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" is_pre_election: true
I20260928 17:47:11.350320 11613 raft_consensus.cc:2454] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f4c0609bfb4148b9d437773f2959eb in term 0.
I20260928 17:47:11.350302 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ec2ac672134f8eb1a922fd38095e3d" candidate_uuid: "02f4c0609bfb4148b9d437773f2959eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e322f70254082820f61a9a68a6316" is_pre_election: true
I20260928 17:47:11.350451 11753 raft_consensus.cc:2454] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f4c0609bfb4148b9d437773f2959eb in term 0.
I20260928 17:47:11.350528 11826 leader_election.cc:302] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f4c0609bfb4148b9d437773f2959eb, 7fd1f5b9293f48caba6fc5c6797d0dec; no voters:
I20260928 17:47:11.350651 11972 raft_consensus.cc:2781] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:47:11.350714 11972 raft_consensus.cc:482] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:47:11.350731 11972 raft_consensus.cc:3037] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.351645 11972 raft_consensus.cc:504] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.351763 11972 leader_election.cc:288] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [CANDIDATE]: Term 1 election: Requested vote from peers 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627), c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833)
I20260928 17:47:11.351933 11613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ec2ac672134f8eb1a922fd38095e3d" candidate_uuid: "02f4c0609bfb4148b9d437773f2959eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec"
I20260928 17:47:11.351956 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ec2ac672134f8eb1a922fd38095e3d" candidate_uuid: "02f4c0609bfb4148b9d437773f2959eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e322f70254082820f61a9a68a6316"
I20260928 17:47:11.352018 11753 raft_consensus.cc:3037] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.352018 11613 raft_consensus.cc:3037] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.352813 11753 raft_consensus.cc:2454] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f4c0609bfb4148b9d437773f2959eb in term 1.
I20260928 17:47:11.352813 11613 raft_consensus.cc:2454] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f4c0609bfb4148b9d437773f2959eb in term 1.
I20260928 17:47:11.352973 11825 leader_election.cc:302] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f4c0609bfb4148b9d437773f2959eb, c37e322f70254082820f61a9a68a6316; no voters:
I20260928 17:47:11.353099 11972 raft_consensus.cc:2781] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:11.353273 11972 raft_consensus.cc:686] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 1 LEADER]: Becoming Leader. State: Replica: 02f4c0609bfb4148b9d437773f2959eb, State: Running, Role: LEADER
I20260928 17:47:11.353358 11972 consensus_queue.cc:237] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.354113 11455 catalog_manager.cc:5495] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3). New cstate: current_term: 1 leader_uuid: "02f4c0609bfb4148b9d437773f2959eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20260928 17:47:11.389216 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.386978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.3.222.3:35919)
0928 17:47:11.387023 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.387024 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.387047 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:47:11.387079 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.387079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.387097 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.387147 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:11.387327 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.387795 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.388584 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.388773 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.388820 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.388979 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.389075 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.389129 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.389144 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.389159 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:47:11.390892 11753 raft_consensus.cc:1261] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Refusing update from remote peer 02f4c0609bfb4148b9d437773f2959eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.390892 11613 raft_consensus.cc:1261] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Refusing update from remote peer 02f4c0609bfb4148b9d437773f2959eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.391191 11972 consensus_queue.cc:1035] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.391369 11977 consensus_queue.cc:1035] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.393321 11980 mvcc.cc:205] Tried to move back new op lower bound from 7334369818175074304 to 7334369818024427520. Current Snapshot: MvccSnapshot[applied={T|T < 7334369818175074304}]
I20260928 17:47:11.395483 11982 mvcc.cc:205] Tried to move back new op lower bound from 7334369818175074304 to 7334369818024427520. Current Snapshot: MvccSnapshot[applied={T|T < 7334369818175074304}]
I20260928 17:47:11.395607 11456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:11.395709 11456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:11.396418 11456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86: Sending DeleteTablet for 3 replicas of tablet 06ec2ac672134f8eb1a922fd38095e3d
I20260928 17:47:11.396729 11593 tablet_service.cc:1514] Processing DeleteTablet for tablet 06ec2ac672134f8eb1a922fd38095e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:52546
I20260928 17:47:11.396791 11873 tablet_service.cc:1514] Processing DeleteTablet for tablet 06ec2ac672134f8eb1a922fd38095e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:35322
I20260928 17:47:11.396857 11732 tablet_service.cc:1514] Processing DeleteTablet for tablet 06ec2ac672134f8eb1a922fd38095e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:43236
I20260928 17:47:11.396953 11456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:47:11.397094 11456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:47:11.397295 11990 tablet_replica.cc:330] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: stopping tablet replica
I20260928 17:47:11.397395 11990 raft_consensus.cc:2229] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:11.397495 11990 raft_consensus.cc:2258] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.398195 11593 tablet_service.cc:1467] Processing CreateTablet for tablet b49a1733263b468f8389e92ca449218d (DEFAULT_TABLE table=test-table [id=6cce505c6acc4f99b41a3fc784154573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.398221 11732 tablet_service.cc:1467] Processing CreateTablet for tablet b49a1733263b468f8389e92ca449218d (DEFAULT_TABLE table=test-table [id=6cce505c6acc4f99b41a3fc784154573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.398312 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49a1733263b468f8389e92ca449218d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.398331 11732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49a1733263b468f8389e92ca449218d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.398427 11990 ts_tablet_manager.cc:1905] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.399267 11973 tablet_bootstrap.cc:492] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: Bootstrap starting.
I20260928 17:47:11.399601 11968 tablet_bootstrap.cc:492] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: Bootstrap starting.
I20260928 17:47:11.399674 11973 tablet_bootstrap.cc:654] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.399904 11968 tablet_bootstrap.cc:654] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.400018 11990 ts_tablet_manager.cc:1918] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:11.400094 11990 log.cc:1196] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/06ec2ac672134f8eb1a922fd38095e3d
I20260928 17:47:11.400313 11873 tablet_service.cc:1467] Processing CreateTablet for tablet b49a1733263b468f8389e92ca449218d (DEFAULT_TABLE table=test-table [id=6cce505c6acc4f99b41a3fc784154573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.400415 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49a1733263b468f8389e92ca449218d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.400411 11990 ts_tablet_manager.cc:1939] T 06ec2ac672134f8eb1a922fd38095e3d P 7fd1f5b9293f48caba6fc5c6797d0dec: Deleting consensus metadata
I20260928 17:47:11.400804 11442 catalog_manager.cc:4841] TS 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627): tablet 06ec2ac672134f8eb1a922fd38095e3d (table test-table [id=aecb25a4ce3348de94f937c96fb4e658]) successfully deleted
I20260928 17:47:11.401006 11973 tablet_bootstrap.cc:492] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: No bootstrap required, opened a new log
I20260928 17:47:11.401057 11973 ts_tablet_manager.cc:1397] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:11.401217 11973 raft_consensus.cc:348] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.401295 11973 raft_consensus.cc:374] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.401315 11973 raft_consensus.cc:729] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37e322f70254082820f61a9a68a6316, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.401356 11973 consensus_queue.cc:260] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.401439 11973 ts_tablet_manager.cc:1428] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:11.401690 11967 tablet_bootstrap.cc:492] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: Bootstrap starting.
I20260928 17:47:11.402030 11968 tablet_bootstrap.cc:492] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: No bootstrap required, opened a new log
I20260928 17:47:11.402074 11967 tablet_bootstrap.cc:654] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.402082 11968 ts_tablet_manager.cc:1397] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:11.402202 11968 raft_consensus.cc:348] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.402277 11968 raft_consensus.cc:374] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.402299 11968 raft_consensus.cc:729] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd1f5b9293f48caba6fc5c6797d0dec, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.402338 11968 consensus_queue.cc:260] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.402635 11968 ts_tablet_manager.cc:1428] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:11.403004 11967 tablet_bootstrap.cc:492] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: No bootstrap required, opened a new log
I20260928 17:47:11.403049 11967 ts_tablet_manager.cc:1397] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:11.403175 11967 raft_consensus.cc:348] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.403262 11967 raft_consensus.cc:374] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.403283 11967 raft_consensus.cc:729] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f4c0609bfb4148b9d437773f2959eb, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.403332 11967 consensus_queue.cc:260] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.403411 11967 ts_tablet_manager.cc:1428] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.407286 11992 tablet_replica.cc:330] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: stopping tablet replica
I20260928 17:47:11.407394 11992 raft_consensus.cc:2229] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:11.407493 11992 raft_consensus.cc:2258] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.407650 11991 tablet_replica.cc:330] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: stopping tablet replica
I20260928 17:47:11.407739 11991 raft_consensus.cc:2229] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:47:11.407829 11991 raft_consensus.cc:2258] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.407893 11992 ts_tablet_manager.cc:1905] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.408180 11991 ts_tablet_manager.cc:1905] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.409536 11991 ts_tablet_manager.cc:1918] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:11.409538 11992 ts_tablet_manager.cc:1918] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:11.409595 11992 log.cc:1196] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/06ec2ac672134f8eb1a922fd38095e3d
I20260928 17:47:11.409595 11991 log.cc:1196] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/06ec2ac672134f8eb1a922fd38095e3d
I20260928 17:47:11.409832 11992 ts_tablet_manager.cc:1939] T 06ec2ac672134f8eb1a922fd38095e3d P c37e322f70254082820f61a9a68a6316: Deleting consensus metadata
I20260928 17:47:11.409843 11991 ts_tablet_manager.cc:1939] T 06ec2ac672134f8eb1a922fd38095e3d P 02f4c0609bfb4148b9d437773f2959eb: Deleting consensus metadata
I20260928 17:47:11.410166 11442 catalog_manager.cc:4841] TS 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919): tablet 06ec2ac672134f8eb1a922fd38095e3d (table test-table [id=aecb25a4ce3348de94f937c96fb4e658]) successfully deleted
I20260928 17:47:11.410279 11441 catalog_manager.cc:4841] TS c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833): tablet 06ec2ac672134f8eb1a922fd38095e3d (table test-table [id=aecb25a4ce3348de94f937c96fb4e658]) successfully deleted
W20260928 17:47:11.412170 11940 tablet.cc:2369] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:47:11.447714 11971 raft_consensus.cc:482] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:11.447818 11971 raft_consensus.cc:504] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.448158 11971 leader_election.cc:288] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833), 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919)
I20260928 17:47:11.451494 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.448303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35919 (local address 127.3.222.1:42081)
0928 17:47:11.448527 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.448536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.448617 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.448790 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.448791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.448793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:11.448868 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.448870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.449476 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.449477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.450721 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.450723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.451260 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.451261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.451324 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.451327 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.451371 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.451408 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":63,"threads_started":1}
I20260928 17:47:11.451481 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.448428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:42081 (local address 127.3.222.3:35919)
0928 17:47:11.448470 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.448472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.448592 (+ 120us) server_negotiation.cc:407] Connection header received
0928 17:47:11.448716 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.448717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.448726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.448777 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:11.448936 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.449405 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.450802 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.451162 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.451248 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.451261 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.451383 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.451415 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:47:11.451767 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.448412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:47991 (local address 127.3.222.2:35833)
0928 17:47:11.448454 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.448457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.448596 (+ 139us) server_negotiation.cc:407] Connection header received
0928 17:47:11.448650 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.448651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.448660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.448710 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:11.449483 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.449968 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.450878 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.451401 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.451466 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.451476 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.451648 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.451701 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:47:11.451792 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49a1733263b468f8389e92ca449218d" candidate_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f4c0609bfb4148b9d437773f2959eb" is_pre_election: true
I20260928 17:47:11.451908 11893 raft_consensus.cc:2454] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd1f5b9293f48caba6fc5c6797d0dec in term 0.
I20260928 17:47:11.451972 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.448311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:35833 (local address 127.3.222.1:47991)
0928 17:47:11.448524 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.448533 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.448590 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.448812 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.448813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.448814 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:11.448875 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.448876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.450044 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.450045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.450791 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.450794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.451508 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.451508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.451579 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.451582 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.451845 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.451883 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"spinlock_wait_cycles":768,"thread_start_us":84,"threads_started":1}
I20260928 17:47:11.452100 11546 leader_election.cc:302] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f4c0609bfb4148b9d437773f2959eb, 7fd1f5b9293f48caba6fc5c6797d0dec; no voters:
I20260928 17:47:11.452217 11971 raft_consensus.cc:2781] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:47:11.452261 11971 raft_consensus.cc:482] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:47:11.452278 11971 raft_consensus.cc:3037] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.452235 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49a1733263b468f8389e92ca449218d" candidate_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e322f70254082820f61a9a68a6316" is_pre_election: true
I20260928 17:47:11.452309 11753 raft_consensus.cc:2454] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd1f5b9293f48caba6fc5c6797d0dec in term 0.
I20260928 17:47:11.452917 11971 raft_consensus.cc:504] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.453033 11971 leader_election.cc:288] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [CANDIDATE]: Term 1 election: Requested vote from peers c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833), 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919)
I20260928 17:47:11.453178 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49a1733263b468f8389e92ca449218d" candidate_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e322f70254082820f61a9a68a6316"
I20260928 17:47:11.453202 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49a1733263b468f8389e92ca449218d" candidate_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f4c0609bfb4148b9d437773f2959eb"
I20260928 17:47:11.453259 11753 raft_consensus.cc:3037] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.453259 11893 raft_consensus.cc:3037] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.453938 11893 raft_consensus.cc:2454] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd1f5b9293f48caba6fc5c6797d0dec in term 1.
I20260928 17:47:11.453938 11753 raft_consensus.cc:2454] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd1f5b9293f48caba6fc5c6797d0dec in term 1.
I20260928 17:47:11.454100 11545 leader_election.cc:302] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd1f5b9293f48caba6fc5c6797d0dec, c37e322f70254082820f61a9a68a6316; no voters:
I20260928 17:47:11.454185 11971 raft_consensus.cc:2781] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:11.454314 11971 raft_consensus.cc:686] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 LEADER]: Becoming Leader. State: Replica: 7fd1f5b9293f48caba6fc5c6797d0dec, State: Running, Role: LEADER
I20260928 17:47:11.454380 11971 consensus_queue.cc:237] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.455044 11456 catalog_manager.cc:5495] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1). New cstate: current_term: 1 leader_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617630, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.62@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:11.483357 12005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.475891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40167 (local address 127.0.0.1:40306)
0928 17:47:11.476019 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.476030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.476085 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.476217 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.476218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.476502 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:11.476550 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.476552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.477506 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.477507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.478133 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.478135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.478605 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.478606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.478644 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.479150 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:11.479154 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:11.480457 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:11.482227 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.482234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.482236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.482424 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.482661 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:11.482662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:11.482662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:11.482752 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:11.483025 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:11.483031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:11.483075 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.483126 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.483275 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260928 17:47:11.483453 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.475967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.3.222.62:40167)
0928 17:47:11.476025 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.476027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.476089 (+ 62us) server_negotiation.cc:407] Connection header received
0928 17:47:11.476123 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.476125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.476137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.476210 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:11.476645 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.477431 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.478217 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.478505 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.478603 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.480549 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:11.480550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:11.480551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:11.480558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:11.482141 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.482497 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.482498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.482499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.482571 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.482825 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.482826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.482827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.482961 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:11.483032 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.483169 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.483343 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":32}
I20260928 17:47:11.485558 11456 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:40306:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 17:47:11.485790 11456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:47:11.487174 11873 tablet_service.cc:1467] Processing CreateTablet for tablet b4dd3615f1b545eab3e82958f735b67b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf30ce05f5db49968e3fc886b396e7dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:47:11.487279 11593 tablet_service.cc:1467] Processing CreateTablet for tablet b4dd3615f1b545eab3e82958f735b67b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf30ce05f5db49968e3fc886b396e7dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:47:11.487357 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4dd3615f1b545eab3e82958f735b67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.487394 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4dd3615f1b545eab3e82958f735b67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.488749 11968 tablet_bootstrap.cc:492] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: Bootstrap starting.
I20260928 17:47:11.489142 11968 tablet_bootstrap.cc:654] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.489118 11732 tablet_service.cc:1467] Processing CreateTablet for tablet b4dd3615f1b545eab3e82958f735b67b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf30ce05f5db49968e3fc886b396e7dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:47:11.489256 11732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4dd3615f1b545eab3e82958f735b67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.489507 11967 tablet_bootstrap.cc:492] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: Bootstrap starting.
I20260928 17:47:11.489894 11967 tablet_bootstrap.cc:654] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.490425 11968 tablet_bootstrap.cc:492] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: No bootstrap required, opened a new log
I20260928 17:47:11.490482 11968 ts_tablet_manager.cc:1397] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:11.490626 11968 raft_consensus.cc:348] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.490712 11968 raft_consensus.cc:374] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.490732 11968 raft_consensus.cc:729] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd1f5b9293f48caba6fc5c6797d0dec, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.490775 11968 consensus_queue.cc:260] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.490870 11968 ts_tablet_manager.cc:1428] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.490963 11973 tablet_bootstrap.cc:492] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: Bootstrap starting.
I20260928 17:47:11.491346 11973 tablet_bootstrap.cc:654] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.491334 11971 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.491421 11971 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:47:11.491734 11967 tablet_bootstrap.cc:492] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: No bootstrap required, opened a new log
I20260928 17:47:11.491777 11967 ts_tablet_manager.cc:1397] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:11.491896 11973 tablet_bootstrap.cc:492] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: No bootstrap required, opened a new log
I20260928 17:47:11.491928 11973 ts_tablet_manager.cc:1397] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:47:11.491905 11967 raft_consensus.cc:348] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.491966 11967 raft_consensus.cc:374] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.491982 11967 raft_consensus.cc:729] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f4c0609bfb4148b9d437773f2959eb, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.492027 11967 consensus_queue.cc:260] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.492044 11973 raft_consensus.cc:348] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.492097 11973 raft_consensus.cc:374] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.492103 11967 ts_tablet_manager.cc:1428] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.492115 11973 raft_consensus.cc:729] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37e322f70254082820f61a9a68a6316, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.492151 11973 consensus_queue.cc:260] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.492141 11987 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.492215 11987 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:47:11.492242 11973 ts_tablet_manager.cc:1428] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.492344 11975 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.492408 11975 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:47:11.505311 11987 raft_consensus.cc:482] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:11.505400 11987 raft_consensus.cc:504] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.505540 11987 leader_election.cc:288] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627), c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833)
I20260928 17:47:11.505702 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dd3615f1b545eab3e82958f735b67b" candidate_uuid: "02f4c0609bfb4148b9d437773f2959eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e322f70254082820f61a9a68a6316" is_pre_election: true
I20260928 17:47:11.505785 11753 raft_consensus.cc:2454] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f4c0609bfb4148b9d437773f2959eb in term 0.
I20260928 17:47:11.505712 11613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dd3615f1b545eab3e82958f735b67b" candidate_uuid: "02f4c0609bfb4148b9d437773f2959eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" is_pre_election: true
I20260928 17:47:11.505825 11613 raft_consensus.cc:2454] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f4c0609bfb4148b9d437773f2959eb in term 0.
I20260928 17:47:11.505925 11825 leader_election.cc:302] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f4c0609bfb4148b9d437773f2959eb, c37e322f70254082820f61a9a68a6316; no voters:
I20260928 17:47:11.506017 11987 raft_consensus.cc:2781] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:47:11.506060 11987 raft_consensus.cc:482] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:47:11.506074 11987 raft_consensus.cc:3037] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.506769 11987 raft_consensus.cc:504] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.506883 11987 leader_election.cc:288] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [CANDIDATE]: Term 1 election: Requested vote from peers 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627), c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833)
I20260928 17:47:11.507033 11613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dd3615f1b545eab3e82958f735b67b" candidate_uuid: "02f4c0609bfb4148b9d437773f2959eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec"
I20260928 17:47:11.507068 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dd3615f1b545eab3e82958f735b67b" candidate_uuid: "02f4c0609bfb4148b9d437773f2959eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e322f70254082820f61a9a68a6316"
I20260928 17:47:11.507113 11613 raft_consensus.cc:3037] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.507131 11753 raft_consensus.cc:3037] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.507809 11613 raft_consensus.cc:2454] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f4c0609bfb4148b9d437773f2959eb in term 1.
I20260928 17:47:11.507809 11753 raft_consensus.cc:2454] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f4c0609bfb4148b9d437773f2959eb in term 1.
I20260928 17:47:11.507961 11826 leader_election.cc:302] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f4c0609bfb4148b9d437773f2959eb, 7fd1f5b9293f48caba6fc5c6797d0dec; no voters:
I20260928 17:47:11.508061 11987 raft_consensus.cc:2781] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:11.508116 11987 raft_consensus.cc:686] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [term 1 LEADER]: Becoming Leader. State: Replica: 02f4c0609bfb4148b9d437773f2959eb, State: Running, Role: LEADER
I20260928 17:47:11.508157 11987 consensus_queue.cc:237] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.508379 11972 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: TxnStatusTablet state changed. Reason: New leader 02f4c0609bfb4148b9d437773f2959eb. Latest consensus state: current_term: 1 leader_uuid: "02f4c0609bfb4148b9d437773f2959eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.508459 11972 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:47:11.508661 12010 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 17:47:11.508641 11456 catalog_manager.cc:5495] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3). New cstate: current_term: 1 leader_uuid: "02f4c0609bfb4148b9d437773f2959eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20260928 17:47:11.508793 12010 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 17:47:11.517522 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.515487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.3.222.3:35919)
0928 17:47:11.515537 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.515539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.515546 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:11.515566 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.515566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.515578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.515621 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:11.515738 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.516204 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.516906 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.517103 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.517153 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.517301 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.517336 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.517379 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.517432 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.517451 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:47:11.517540 12005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.515311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35919 (local address 127.0.0.1:35346)
0928 17:47:11.515374 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.515385 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.515422 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.515638 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.515639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.515640 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:47:11.515687 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.515689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.516274 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.516275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.516834 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.516836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.517158 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.517159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.517224 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.517237 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.517384 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.517390 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.517431 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.517477 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 17:47:11.518368 11753 raft_consensus.cc:1261] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Refusing update from remote peer 02f4c0609bfb4148b9d437773f2959eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.518384 11613 raft_consensus.cc:1261] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Refusing update from remote peer 02f4c0609bfb4148b9d437773f2959eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.518546 11972 consensus_queue.cc:1035] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.518621 11987 consensus_queue.cc:1035] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.518999 11975 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: TxnStatusTablet state changed. Reason: New leader 02f4c0609bfb4148b9d437773f2959eb. Latest consensus state: current_term: 1 leader_uuid: "02f4c0609bfb4148b9d437773f2959eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.519088 11975 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:47:11.519106 11971 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: TxnStatusTablet state changed. Reason: New leader 02f4c0609bfb4148b9d437773f2959eb. Latest consensus state: current_term: 1 leader_uuid: "02f4c0609bfb4148b9d437773f2959eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.519179 11971 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:47:11.519737 11972 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02f4c0609bfb4148b9d437773f2959eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.519809 11972 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:47:11.520032 11981 mvcc.cc:205] Tried to move back new op lower bound from 7334369818698702848 to 7334369818658410496. Current Snapshot: MvccSnapshot[applied={T|T < 7334369818698702848 or (T in {7334369818698702848})}]
I20260928 17:47:11.520195 11971 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02f4c0609bfb4148b9d437773f2959eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.520267 11971 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P 7fd1f5b9293f48caba6fc5c6797d0dec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:47:11.520689 11977 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02f4c0609bfb4148b9d437773f2959eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.520761 11977 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P 02f4c0609bfb4148b9d437773f2959eb: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:47:11.521293 11975 tablet_replica.cc:439] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02f4c0609bfb4148b9d437773f2959eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } } }
I20260928 17:47:11.521369 11975 tablet_replica.cc:442] T b4dd3615f1b545eab3e82958f735b67b P c37e322f70254082820f61a9a68a6316: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:47:11.524129 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.521656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.3.222.1:44627)
0928 17:47:11.521712 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.521713 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.521723 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:11.521777 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.521778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.521796 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.521866 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:11.521961 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.522661 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.523398 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.523696 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.523756 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.523845 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.523963 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.524032 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.524046 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.524065 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:47:11.528648 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.526371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:32999 (local address 127.3.222.3:35919)
0928 17:47:11.526421 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.526422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.526429 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:11.526489 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.526490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.526500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.526543 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:11.526683 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.527109 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.528033 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.528230 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.528304 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.528438 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.528469 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.528533 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.528568 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.528588 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:47:11.528858 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.526246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35919 (local address 127.3.222.1:32999)
0928 17:47:11.526377 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.526387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.526436 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.526549 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.526549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.526569 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:47:11.526628 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.526629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.527168 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.527169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.527963 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.527965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.528292 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.528293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.528355 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.528369 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.528518 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.528520 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.528723 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.528767 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260928 17:47:11.530689 11753 raft_consensus.cc:1261] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Refusing update from remote peer 7fd1f5b9293f48caba6fc5c6797d0dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.530714 11893 raft_consensus.cc:1261] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Refusing update from remote peer 7fd1f5b9293f48caba6fc5c6797d0dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.530905 11998 consensus_queue.cc:1035] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.531021 11971 consensus_queue.cc:1035] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.538151 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.535901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:39779 (local address 127.3.222.1:44627)
0928 17:47:11.535929 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.535931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.535938 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:11.535956 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.535956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.535965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.536012 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:11.536095 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.536774 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.537527 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.537783 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.537843 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.537989 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.538012 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.538040 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.538091 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.538103 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:47:11.538196 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.535815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44627 (local address 127.3.222.3:39779)
0928 17:47:11.535864 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.535874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.535913 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.536018 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.536019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.536020 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:47:11.536052 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.536053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.536844 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.536845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.537464 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.537467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.537842 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.537843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.537905 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.537919 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.538043 (+ 124us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.538045 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.538090 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.538128 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 17:47:11.544461 11455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:11.544544 11455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:11.545126 11455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86: Sending DeleteTablet for 3 replicas of tablet b49a1733263b468f8389e92ca449218d
I20260928 17:47:11.545351 11593 tablet_service.cc:1514] Processing DeleteTablet for tablet b49a1733263b468f8389e92ca449218d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:52546
I20260928 17:47:11.545426 11990 tablet_replica.cc:330] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: stopping tablet replica
I20260928 17:47:11.545414 11732 tablet_service.cc:1514] Processing DeleteTablet for tablet b49a1733263b468f8389e92ca449218d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:43236
I20260928 17:47:11.545465 11990 raft_consensus.cc:2229] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:47:11.545488 11992 tablet_replica.cc:330] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: stopping tablet replica
I20260928 17:47:11.545528 11992 raft_consensus.cc:2229] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:11.545532 11990 raft_consensus.cc:2258] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.545557 11992 raft_consensus.cc:2258] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.545722 11873 tablet_service.cc:1514] Processing DeleteTablet for tablet b49a1733263b468f8389e92ca449218d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:35322
I20260928 17:47:11.545835 11990 ts_tablet_manager.cc:1905] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.545998 11992 ts_tablet_manager.cc:1905] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.547087 11991 tablet_replica.cc:330] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: stopping tablet replica
I20260928 17:47:11.547148 11991 raft_consensus.cc:2229] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:11.547183 11991 raft_consensus.cc:2258] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.547492 11991 ts_tablet_manager.cc:1905] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.548949 11991 ts_tablet_manager.cc:1918] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:47:11.548995 11991 log.cc:1196] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/b49a1733263b468f8389e92ca449218d
I20260928 17:47:11.549197 11991 ts_tablet_manager.cc:1939] T b49a1733263b468f8389e92ca449218d P 02f4c0609bfb4148b9d437773f2959eb: Deleting consensus metadata
I20260928 17:47:11.549459 11990 ts_tablet_manager.cc:1918] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:47:11.549507 11990 log.cc:1196] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/b49a1733263b468f8389e92ca449218d
I20260928 17:47:11.549688 11990 ts_tablet_manager.cc:1939] T b49a1733263b468f8389e92ca449218d P 7fd1f5b9293f48caba6fc5c6797d0dec: Deleting consensus metadata
I20260928 17:47:11.550061 11992 ts_tablet_manager.cc:1918] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:47:11.550125 11992 log.cc:1196] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/b49a1733263b468f8389e92ca449218d
I20260928 17:47:11.550194 11442 catalog_manager.cc:4841] TS 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627): tablet b49a1733263b468f8389e92ca449218d (table test-table [id=6cce505c6acc4f99b41a3fc784154573]) successfully deleted
I20260928 17:47:11.550309 11992 ts_tablet_manager.cc:1939] T b49a1733263b468f8389e92ca449218d P c37e322f70254082820f61a9a68a6316: Deleting consensus metadata
I20260928 17:47:11.550563 11441 catalog_manager.cc:4841] TS c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833): tablet b49a1733263b468f8389e92ca449218d (table test-table [id=6cce505c6acc4f99b41a3fc784154573]) successfully deleted
I20260928 17:47:11.550721 11442 catalog_manager.cc:4841] TS 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919): tablet b49a1733263b468f8389e92ca449218d (table test-table [id=6cce505c6acc4f99b41a3fc784154573]) successfully deleted
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 17:47:11.585471 11506 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:11.578779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.3.222.62:40167)
0928 17:47:11.578824 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.578826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.578978 (+ 152us) server_negotiation.cc:407] Connection header received
0928 17:47:11.579052 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.579053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.579066 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.579154 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:11.579903 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.580651 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.581722 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.582015 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.582084 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.585400 (+ 3316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40318: BlockingRecv error: recv got EOF from 127.0.0.1:40318 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[11411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM
I20260928 17:47:11.595021 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.586508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.3.222.62:40167)
0928 17:47:11.586559 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.586560 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.586850 (+ 290us) server_negotiation.cc:407] Connection header received
0928 17:47:11.586906 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.586907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.586917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.586974 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:11.587571 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.588306 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.589301 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.589605 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.589673 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.592205 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:11.592207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:11.592207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:11.592216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:11.593852 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.594212 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.594213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.594214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.594312 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:11.594503 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:11.594504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:11.594504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:11.594645 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:11.594746 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.594841 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.594922 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:11.596740 11455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:47:11.596912 11455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:47:11.598196 11593 tablet_service.cc:1467] Processing CreateTablet for tablet 46c57a9c78eb4f3f879ebd7f7c1d680f (DEFAULT_TABLE table=test-table [id=6b09c25752704713a2565f29d310ac6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.598208 11873 tablet_service.cc:1467] Processing CreateTablet for tablet 46c57a9c78eb4f3f879ebd7f7c1d680f (DEFAULT_TABLE table=test-table [id=6b09c25752704713a2565f29d310ac6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.598330 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46c57a9c78eb4f3f879ebd7f7c1d680f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.598345 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46c57a9c78eb4f3f879ebd7f7c1d680f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.599674 11732 tablet_service.cc:1467] Processing CreateTablet for tablet 46c57a9c78eb4f3f879ebd7f7c1d680f (DEFAULT_TABLE table=test-table [id=6b09c25752704713a2565f29d310ac6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.599749 11967 tablet_bootstrap.cc:492] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: Bootstrap starting.
I20260928 17:47:11.599779 11732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46c57a9c78eb4f3f879ebd7f7c1d680f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.600068 11967 tablet_bootstrap.cc:654] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.600915 11968 tablet_bootstrap.cc:492] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: Bootstrap starting.
I20260928 17:47:11.600970 11967 tablet_bootstrap.cc:492] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: No bootstrap required, opened a new log
I20260928 17:47:11.601012 11967 ts_tablet_manager.cc:1397] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:11.601137 11967 raft_consensus.cc:348] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.601202 11967 raft_consensus.cc:374] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.601219 11967 raft_consensus.cc:729] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f4c0609bfb4148b9d437773f2959eb, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.601274 11967 consensus_queue.cc:260] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.601305 11968 tablet_bootstrap.cc:654] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.601358 11967 ts_tablet_manager.cc:1428] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.601804 11973 tablet_bootstrap.cc:492] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: Bootstrap starting.
I20260928 17:47:11.602200 11973 tablet_bootstrap.cc:654] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.603389 11973 tablet_bootstrap.cc:492] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: No bootstrap required, opened a new log
I20260928 17:47:11.603439 11973 ts_tablet_manager.cc:1397] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:47:11.603574 11973 raft_consensus.cc:348] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.603644 11973 raft_consensus.cc:374] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.603662 11973 raft_consensus.cc:729] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37e322f70254082820f61a9a68a6316, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.603700 11973 consensus_queue.cc:260] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.603780 11973 ts_tablet_manager.cc:1428] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:11.604079 11968 tablet_bootstrap.cc:492] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: No bootstrap required, opened a new log
I20260928 17:47:11.604125 11968 ts_tablet_manager.cc:1397] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 17:47:11.604252 11968 raft_consensus.cc:348] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.604316 11968 raft_consensus.cc:374] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.604334 11968 raft_consensus.cc:729] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd1f5b9293f48caba6fc5c6797d0dec, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.604368 11968 consensus_queue.cc:260] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.604441 11968 ts_tablet_manager.cc:1428] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.665297 11998 raft_consensus.cc:482] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:11.665403 11998 raft_consensus.cc:504] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.665572 11998 leader_election.cc:288] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833), 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919)
I20260928 17:47:11.665750 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c57a9c78eb4f3f879ebd7f7c1d680f" candidate_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f4c0609bfb4148b9d437773f2959eb" is_pre_election: true
I20260928 17:47:11.665763 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c57a9c78eb4f3f879ebd7f7c1d680f" candidate_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e322f70254082820f61a9a68a6316" is_pre_election: true
I20260928 17:47:11.665851 11753 raft_consensus.cc:2454] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd1f5b9293f48caba6fc5c6797d0dec in term 0.
I20260928 17:47:11.665851 11893 raft_consensus.cc:2454] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd1f5b9293f48caba6fc5c6797d0dec in term 0.
I20260928 17:47:11.666029 11546 leader_election.cc:302] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f4c0609bfb4148b9d437773f2959eb, 7fd1f5b9293f48caba6fc5c6797d0dec; no voters:
I20260928 17:47:11.666128 11998 raft_consensus.cc:2781] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:47:11.666167 11998 raft_consensus.cc:482] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:47:11.666189 11998 raft_consensus.cc:3037] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.666783 11998 raft_consensus.cc:504] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.666883 11998 leader_election.cc:288] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [CANDIDATE]: Term 1 election: Requested vote from peers c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833), 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919)
I20260928 17:47:11.667042 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c57a9c78eb4f3f879ebd7f7c1d680f" candidate_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e322f70254082820f61a9a68a6316"
I20260928 17:47:11.667100 11753 raft_consensus.cc:3037] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.667055 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c57a9c78eb4f3f879ebd7f7c1d680f" candidate_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f4c0609bfb4148b9d437773f2959eb"
I20260928 17:47:11.667130 11893 raft_consensus.cc:3037] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.667660 11753 raft_consensus.cc:2454] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd1f5b9293f48caba6fc5c6797d0dec in term 1.
I20260928 17:47:11.667764 11893 raft_consensus.cc:2454] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd1f5b9293f48caba6fc5c6797d0dec in term 1.
I20260928 17:47:11.667835 11545 leader_election.cc:302] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd1f5b9293f48caba6fc5c6797d0dec, c37e322f70254082820f61a9a68a6316; no voters:
I20260928 17:47:11.667922 11998 raft_consensus.cc:2781] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:11.667981 11998 raft_consensus.cc:686] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 LEADER]: Becoming Leader. State: Replica: 7fd1f5b9293f48caba6fc5c6797d0dec, State: Running, Role: LEADER
I20260928 17:47:11.668035 11998 consensus_queue.cc:237] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.668550 11455 catalog_manager.cc:5495] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1). New cstate: current_term: 1 leader_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20260928 17:47:11.678321 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.676150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.3.222.1:44627)
0928 17:47:11.676204 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.676206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.676215 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:11.676275 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.676277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.676286 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.676327 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:11.676454 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.676934 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.677709 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.677894 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.677942 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.678111 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.678138 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.678179 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.678250 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.678267 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:47:11.678946 11753 raft_consensus.cc:1261] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Refusing update from remote peer 7fd1f5b9293f48caba6fc5c6797d0dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.678946 11893 raft_consensus.cc:1261] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Refusing update from remote peer 7fd1f5b9293f48caba6fc5c6797d0dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.679143 11998 consensus_queue.cc:1035] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.679261 11971 consensus_queue.cc:1035] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.681406 11456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:11.681481 11456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:11.681963 11456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86: Sending DeleteTablet for 3 replicas of tablet 46c57a9c78eb4f3f879ebd7f7c1d680f
I20260928 17:47:11.682183 11593 tablet_service.cc:1514] Processing DeleteTablet for tablet 46c57a9c78eb4f3f879ebd7f7c1d680f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:52546
I20260928 17:47:11.682227 11732 tablet_service.cc:1514] Processing DeleteTablet for tablet 46c57a9c78eb4f3f879ebd7f7c1d680f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:43236
I20260928 17:47:11.682266 11990 tablet_replica.cc:330] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: stopping tablet replica
I20260928 17:47:11.682312 11990 raft_consensus.cc:2229] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:47:11.682314 11873 tablet_service.cc:1514] Processing DeleteTablet for tablet 46c57a9c78eb4f3f879ebd7f7c1d680f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:35322
I20260928 17:47:11.682367 11990 raft_consensus.cc:2258] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.682394 11992 tablet_replica.cc:330] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: stopping tablet replica
I20260928 17:47:11.682363 11456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 17:47:11.682451 11992 raft_consensus.cc:2229] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 17:47:11.682480 11456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:47:11.682493 11992 raft_consensus.cc:2258] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.682559 11991 tablet_replica.cc:330] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: stopping tablet replica
I20260928 17:47:11.682611 11991 raft_consensus.cc:2229] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:11.682644 11991 raft_consensus.cc:2258] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.682857 11990 ts_tablet_manager.cc:1905] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.682920 11991 ts_tablet_manager.cc:1905] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.682945 11992 ts_tablet_manager.cc:1905] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.684392 11992 ts_tablet_manager.cc:1918] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:11.684392 11990 ts_tablet_manager.cc:1918] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:11.684444 11990 log.cc:1196] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/46c57a9c78eb4f3f879ebd7f7c1d680f
I20260928 17:47:11.684444 11992 log.cc:1196] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/46c57a9c78eb4f3f879ebd7f7c1d680f
I20260928 17:47:11.684638 11992 ts_tablet_manager.cc:1939] T 46c57a9c78eb4f3f879ebd7f7c1d680f P c37e322f70254082820f61a9a68a6316: Deleting consensus metadata
I20260928 17:47:11.684638 11990 ts_tablet_manager.cc:1939] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 7fd1f5b9293f48caba6fc5c6797d0dec: Deleting consensus metadata
I20260928 17:47:11.684780 11732 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf5e04e46764825955c1b62341f787a (DEFAULT_TABLE table=test-table [id=3b8a37a70f4046b3800673ce050e062c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.684900 11732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf5e04e46764825955c1b62341f787a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.684954 11442 catalog_manager.cc:4841] TS 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627): tablet 46c57a9c78eb4f3f879ebd7f7c1d680f (table test-table [id=6b09c25752704713a2565f29d310ac6a]) successfully deleted
I20260928 17:47:11.685002 11593 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf5e04e46764825955c1b62341f787a (DEFAULT_TABLE table=test-table [id=3b8a37a70f4046b3800673ce050e062c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.685079 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf5e04e46764825955c1b62341f787a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.685103 11991 ts_tablet_manager.cc:1918] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:11.685137 11991 log.cc:1196] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/46c57a9c78eb4f3f879ebd7f7c1d680f
I20260928 17:47:11.685313 11991 ts_tablet_manager.cc:1939] T 46c57a9c78eb4f3f879ebd7f7c1d680f P 02f4c0609bfb4148b9d437773f2959eb: Deleting consensus metadata
I20260928 17:47:11.685531 11442 catalog_manager.cc:4841] TS 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919): tablet 46c57a9c78eb4f3f879ebd7f7c1d680f (table test-table [id=6b09c25752704713a2565f29d310ac6a]) successfully deleted
I20260928 17:47:11.685925 11973 tablet_bootstrap.cc:492] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: Bootstrap starting.
I20260928 17:47:11.686146 11968 tablet_bootstrap.cc:492] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec: Bootstrap starting.
I20260928 17:47:11.686174 11873 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf5e04e46764825955c1b62341f787a (DEFAULT_TABLE table=test-table [id=3b8a37a70f4046b3800673ce050e062c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:11.686259 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf5e04e46764825955c1b62341f787a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:11.686427 11968 tablet_bootstrap.cc:654] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.686570 11973 tablet_bootstrap.cc:654] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.686918 11968 tablet_bootstrap.cc:492] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec: No bootstrap required, opened a new log
I20260928 17:47:11.686946 11968 ts_tablet_manager.cc:1397] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:11.687014 11968 raft_consensus.cc:348] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.687054 11968 raft_consensus.cc:374] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.687065 11968 raft_consensus.cc:729] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd1f5b9293f48caba6fc5c6797d0dec, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.687100 11968 consensus_queue.cc:260] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.687127 11967 tablet_bootstrap.cc:492] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: Bootstrap starting.
I20260928 17:47:11.687177 11968 ts_tablet_manager.cc:1428] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.687342 11441 catalog_manager.cc:4841] TS c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833): tablet 46c57a9c78eb4f3f879ebd7f7c1d680f (table test-table [id=6b09c25752704713a2565f29d310ac6a]) successfully deleted
I20260928 17:47:11.687404 11967 tablet_bootstrap.cc:654] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:11.687525 11973 tablet_bootstrap.cc:492] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: No bootstrap required, opened a new log
I20260928 17:47:11.687561 11973 ts_tablet_manager.cc:1397] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:47:11.687685 11973 raft_consensus.cc:348] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.687748 11973 raft_consensus.cc:374] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.687772 11973 raft_consensus.cc:729] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37e322f70254082820f61a9a68a6316, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.687811 11973 consensus_queue.cc:260] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.687892 11973 ts_tablet_manager.cc:1428] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.688007 11967 tablet_bootstrap.cc:492] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: No bootstrap required, opened a new log
I20260928 17:47:11.688055 11967 ts_tablet_manager.cc:1397] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 17:47:11.688177 11967 raft_consensus.cc:348] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.688243 11967 raft_consensus.cc:374] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:11.688261 11967 raft_consensus.cc:729] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f4c0609bfb4148b9d437773f2959eb, State: Initialized, Role: FOLLOWER
I20260928 17:47:11.688294 11967 consensus_queue.cc:260] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.688369 11967 ts_tablet_manager.cc:1428] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:11.799518 11975 raft_consensus.cc:482] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:11.799612 11975 raft_consensus.cc:504] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.799943 11975 leader_election.cc:288] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627), 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919)
I20260928 17:47:11.803963 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.800155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:53875 (local address 127.3.222.1:44627)
0928 17:47:11.800240 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.800243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.800305 (+ 62us) server_negotiation.cc:407] Connection header received
0928 17:47:11.800407 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.800409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.800418 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.800446 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:11.800672 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.801455 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.802979 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.803580 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.803655 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.803665 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.803862 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.803897 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 17:47:11.803993 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.800347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:39209 (local address 127.3.222.3:35919)
0928 17:47:11.800406 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.800408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.800416 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:11.800487 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.800488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.800497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.800541 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:11.800671 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.801403 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.802980 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.803581 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.803656 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.803665 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.803860 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.803903 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:47:11.804000 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.800091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44627 (local address 127.3.222.2:53875)
0928 17:47:11.800219 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.800232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.800335 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.800510 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.800511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.800513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:11.800585 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.800587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.801546 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.801547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.802848 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.802850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.803691 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.803692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.803775 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.803783 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.803842 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.803886 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":8,"thread_start_us":71,"threads_started":1}
I20260928 17:47:11.804011 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.800265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35919 (local address 127.3.222.2:39209)
0928 17:47:11.800348 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.800357 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.800408 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.800559 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.800560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.800562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:11.800608 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.800609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.801533 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.801534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.802848 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.802850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.803691 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.803692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.803775 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.803781 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.803847 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.803886 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":61,"thread_start_us":34,"threads_started":1}
I20260928 17:47:11.804270 11613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5e04e46764825955c1b62341f787a" candidate_uuid: "c37e322f70254082820f61a9a68a6316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" is_pre_election: true
I20260928 17:47:11.804281 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5e04e46764825955c1b62341f787a" candidate_uuid: "c37e322f70254082820f61a9a68a6316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f4c0609bfb4148b9d437773f2959eb" is_pre_election: true
I20260928 17:47:11.804356 11893 raft_consensus.cc:2454] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37e322f70254082820f61a9a68a6316 in term 0.
I20260928 17:47:11.804358 11613 raft_consensus.cc:2454] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37e322f70254082820f61a9a68a6316 in term 0.
I20260928 17:47:11.804504 11686 leader_election.cc:302] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd1f5b9293f48caba6fc5c6797d0dec, c37e322f70254082820f61a9a68a6316; no voters:
I20260928 17:47:11.804647 11975 raft_consensus.cc:2781] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:47:11.804690 11975 raft_consensus.cc:482] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:47:11.804703 11975 raft_consensus.cc:3037] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.805338 11975 raft_consensus.cc:504] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.805436 11975 leader_election.cc:288] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [CANDIDATE]: Term 1 election: Requested vote from peers 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627), 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919)
I20260928 17:47:11.805665 11613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5e04e46764825955c1b62341f787a" candidate_uuid: "c37e322f70254082820f61a9a68a6316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec"
I20260928 17:47:11.805735 11613 raft_consensus.cc:3037] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.805697 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5e04e46764825955c1b62341f787a" candidate_uuid: "c37e322f70254082820f61a9a68a6316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f4c0609bfb4148b9d437773f2959eb"
I20260928 17:47:11.805763 11893 raft_consensus.cc:3037] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:11.806317 11613 raft_consensus.cc:2454] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37e322f70254082820f61a9a68a6316 in term 1.
I20260928 17:47:11.806396 11893 raft_consensus.cc:2454] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37e322f70254082820f61a9a68a6316 in term 1.
I20260928 17:47:11.806481 11686 leader_election.cc:302] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd1f5b9293f48caba6fc5c6797d0dec, c37e322f70254082820f61a9a68a6316; no voters:
I20260928 17:47:11.806620 11975 raft_consensus.cc:2781] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:11.806748 11975 raft_consensus.cc:686] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 1 LEADER]: Becoming Leader. State: Replica: c37e322f70254082820f61a9a68a6316, State: Running, Role: LEADER
I20260928 17:47:11.806813 11975 consensus_queue.cc:237] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } }
I20260928 17:47:11.807718 11455 catalog_manager.cc:5495] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37e322f70254082820f61a9a68a6316 (127.3.222.2). New cstate: current_term: 1 leader_uuid: "c37e322f70254082820f61a9a68a6316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37e322f70254082820f61a9a68a6316" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20260928 17:47:11.836090 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.833833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.3.222.2:35833)
0928 17:47:11.833882 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.833884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.833894 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:11.833945 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.833946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.833969 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.834030 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:11.834179 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.834656 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.835379 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.835593 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.835649 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.835788 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.835898 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.835960 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.835997 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.836016 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:47:11.840513 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.838345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:44659 (local address 127.3.222.3:35919)
0928 17:47:11.838393 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.838395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.838439 (+ 44us) server_negotiation.cc:407] Connection header received
0928 17:47:11.838509 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.838509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.838517 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.838562 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:11.838641 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.839098 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.839820 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.840034 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.840094 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.840291 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.840321 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.840414 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.840441 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.840462 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:47:11.840524 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.838269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35919 (local address 127.3.222.2:44659)
0928 17:47:11.838388 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.838397 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.838463 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.838564 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.838564 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.838566 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:47:11.838609 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.838610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.839157 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.839157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.839738 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.839740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.840123 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.840124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.840181 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.840192 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.840380 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.840381 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.840405 (+ 24us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.840448 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":70,"threads_started":1}
I20260928 17:47:11.842399 11893 raft_consensus.cc:1261] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Refusing update from remote peer c37e322f70254082820f61a9a68a6316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.842697 12047 consensus_queue.cc:1035] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f4c0609bfb4148b9d437773f2959eb" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.845063 11613 raft_consensus.cc:1261] T 0bf5e04e46764825955c1b62341f787a P 7fd1f5b9293f48caba6fc5c6797d0dec [term 1 FOLLOWER]: Refusing update from remote peer c37e322f70254082820f61a9a68a6316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:11.845249 11975 consensus_queue.cc:1035] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd1f5b9293f48caba6fc5c6797d0dec" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:11.849509 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.846679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:35833 (local address 127.3.222.3:45055)
0928 17:47:11.846759 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.846769 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:11.846803 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:11.847156 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:11.847157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:11.847159 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:47:11.847197 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.847199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.847924 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.847925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.848571 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:11.848574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.849092 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.849093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:11.849140 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.849145 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.849347 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.849348 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:47:11.849398 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 17:47:11.849444 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 17:47:11.849514 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.846932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:45055 (local address 127.3.222.2:35833)
0928 17:47:11.846965 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:11.846966 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:11.846973 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:11.846990 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:11.846990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:11.846998 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:11.847054 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:11.847288 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.847859 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.848721 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:11.848951 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:11.849052 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:11.849248 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:11.849289 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:11.849344 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:11.849403 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 17:47:11.849433 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:47:11.857287 11455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:11.857375 11455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:11.858011 11455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8139ccf094a4a89a41d2c990b02ad86: Sending DeleteTablet for 3 replicas of tablet 0bf5e04e46764825955c1b62341f787a
I20260928 17:47:11.858229 11732 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf5e04e46764825955c1b62341f787a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:43236
I20260928 17:47:11.858256 11873 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf5e04e46764825955c1b62341f787a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:11 UTC) from {username='oryx', principal='oryx/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:35322
I20260928 17:47:11.858311 11992 tablet_replica.cc:330] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: stopping tablet replica
I20260928 17:47:11.858358 11992 raft_consensus.cc:2229] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:47:11.858412 11991 tablet_replica.cc:330] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: stopping tablet replica
I20260928 17:47:11.858431 11992 raft_consensus.cc:2258] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.858474 11991 raft_consensus.cc:2229] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:11.858515 11991 raft_consensus.cc:2258] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:11.858714 11992 ts_tablet_manager.cc:1905] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.859020 11991 ts_tablet_manager.cc:1905] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:11.859332 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 11528
I20260928 17:47:11.860421 11992 ts_tablet_manager.cc:1918] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:47:11.860476 11992 log.cc:1196] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/0bf5e04e46764825955c1b62341f787a
I20260928 17:47:11.860677 11992 ts_tablet_manager.cc:1939] T 0bf5e04e46764825955c1b62341f787a P c37e322f70254082820f61a9a68a6316: Deleting consensus metadata
I20260928 17:47:11.860948 11441 catalog_manager.cc:4841] TS c37e322f70254082820f61a9a68a6316 (127.3.222.2:35833): tablet 0bf5e04e46764825955c1b62341f787a (table test-table [id=3b8a37a70f4046b3800673ce050e062c]) successfully deleted
I20260928 17:47:11.863328 11991 ts_tablet_manager.cc:1918] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:47:11.863376 11991 log.cc:1196] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/0bf5e04e46764825955c1b62341f787a
I20260928 17:47:11.863560 11991 ts_tablet_manager.cc:1939] T 0bf5e04e46764825955c1b62341f787a P 02f4c0609bfb4148b9d437773f2959eb: Deleting consensus metadata
I20260928 17:47:11.867012 11442 catalog_manager.cc:4841] TS 02f4c0609bfb4148b9d437773f2959eb (127.3.222.3:35919): tablet 0bf5e04e46764825955c1b62341f787a (table test-table [id=3b8a37a70f4046b3800673ce050e062c]) successfully deleted
W20260928 17:47:11.867238 11442 connection.cc:535] client connection to 127.3.222.1:44627 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 17:47:11.867362 11442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:47:11.867765 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:11.867640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44627 (local address 127.0.0.1:52578)
0928 17:47:11.867705 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:11.867735 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51,"thread_start_us":37,"threads_started":1}
W20260928 17:47:11.867846 11442 catalog_manager.cc:4584] TS 7fd1f5b9293f48caba6fc5c6797d0dec (127.3.222.1:44627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bf5e04e46764825955c1b62341f787a: Network error: Client connection negotiation failed: client connection to 127.3.222.1:44627: connect: Connection refused (error 111)
I20260928 17:47:11.868149 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 11668
I20260928 17:47:11.872815 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 11808
I20260928 17:47:11.878568 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 11424
2026-09-28T17:47:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2376 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[12061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[12061](info): set up 2 sockets
Sep 28 17:47:11 dist-test-slave-w3bm krb5kdc[12061](info): commencing operation
krb5kdc: starting...
W20260928 17:47:12.835366 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:47:12 dist-test-slave-w3bm krb5kdc[12061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:47:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:47:12.945916 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41777
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:33985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:13.025923 12074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:13.026109 12074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:13.026129 12074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:13.027225 12074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:13.027256 12074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:13.027267 12074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:13.027278 12074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:13.027288 12074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:13.028743 12074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:13.028923 12074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:13.029090 12074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:47:13.033601 12074 server_base.cc:1034] running on GCE node
W20260928 17:47:13.033564 12080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:13.033571 12079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:13.033562 12082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:13.033939 12074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:13.034211 12074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:13.035332 12074 hybrid_clock.cc:648] HybridClock initialized: now 1790617633035319 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T17:47:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1164 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:47:13 dist-test-slave-w3bm krb5kdc[12087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:47:13 dist-test-slave-w3bm krb5kdc[12087](info): set up 2 sockets
Sep 28 17:47:13 dist-test-slave-w3bm krb5kdc[12087](info): commencing operation
krb5kdc: starting...
W20260928 17:47:14.003460 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:47:14.117664 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:46647
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:46647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:14.195327 12100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:14.195540 12100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:14.195570 12100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:14.196755 12100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:14.196797 12100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:14.196815 12100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:14.196832 12100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:14.196846 12100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:14.198295 12100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:46647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:46647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:14.198541 12100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:14.198740 12100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:14.203299 12106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:14.203398 12100 server_base.cc:1034] running on GCE node
W20260928 17:47:14.203286 12108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:14.203429 12105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:14.203742 12100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:14.204033 12100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:14.205183 12100 hybrid_clock.cc:648] HybridClock initialized: now 1790617634205161 us; error 32 us; skew 500 ppm
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:14.207448 12100 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:47:14.207845 12100 webserver.cc:460] Webserver started at http://127.3.222.62:46059/ using document root <none> and password file <none>
I20260928 17:47:14.208019 12100 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:14.208055 12100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:14.208141 12100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:14.209107 12100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "69f941d8b9124a3e9f0262e90b74a31a"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "4fa8414ff3c5263634dfad7f951a9e23"
server_key_iv: "754d751cb76ee23878a46f3c545b490e"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.209383 12100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "69f941d8b9124a3e9f0262e90b74a31a"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "4fa8414ff3c5263634dfad7f951a9e23"
server_key_iv: "754d751cb76ee23878a46f3c545b490e"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.210537 12100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:14.211191 12115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.211374 12100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:14.211455 12100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "69f941d8b9124a3e9f0262e90b74a31a"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "4fa8414ff3c5263634dfad7f951a9e23"
server_key_iv: "754d751cb76ee23878a46f3c545b490e"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.211521 12100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:14.234650 12100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:14.236672 12100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:14.236799 12100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:14.240836 12100 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:46647
I20260928 17:47:14.240865 12167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:46647 every 8 connection(s)
I20260928 17:47:14.241178 12100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:14.241969 12168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:14.242043 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 12100
I20260928 17:47:14.242146 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:14.242424 3960 external_mini_cluster.cc:1428] Setting key 65826b65d9ef0c1c1ef58755bf30b409
I20260928 17:47:14.244856 12168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: Bootstrap starting.
I20260928 17:47:14.245556 12168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:14.245823 12168 log.cc:824] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:14.246595 12168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: No bootstrap required, opened a new log
I20260928 17:47:14.248593 12168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } }
I20260928 17:47:14.248732 12168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:14.248759 12168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f941d8b9124a3e9f0262e90b74a31a, State: Initialized, Role: FOLLOWER
I20260928 17:47:14.248919 12168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } }
I20260928 17:47:14.249007 12168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:14.249042 12173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:14.249130 12173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } }
I20260928 17:47:14.249240 12173 leader_election.cc:302] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f941d8b9124a3e9f0262e90b74a31a; no voters:
I20260928 17:47:14.249265 12168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:14.249312 12168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:14.249413 12173 leader_election.cc:288] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260928 17:47:14.249869 12168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } }
I20260928 17:47:14.249940 12168 leader_election.cc:302] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f941d8b9124a3e9f0262e90b74a31a; no voters:
I20260928 17:47:14.249975 12168 leader_election.cc:288] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:14.249994 12174 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 17:47:14.250113 12173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:14.250157 12173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 1 LEADER]: Becoming Leader. State: Replica: 69f941d8b9124a3e9f0262e90b74a31a, State: Running, Role: LEADER
I20260928 17:47:14.250221 12168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:14.250228 12173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } }
I20260928 17:47:14.250542 12173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f941d8b9124a3e9f0262e90b74a31a. Latest consensus state: current_term: 1 leader_uuid: "69f941d8b9124a3e9f0262e90b74a31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } } }
I20260928 17:47:14.250631 12173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:14.250552 12174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69f941d8b9124a3e9f0262e90b74a31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } } }
I20260928 17:47:14.250808 12174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:14.250890 12180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:14.251507 12180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:14.252835 12180 catalog_manager.cc:1347] Generated new cluster ID: 67f2d76c9f8d47ebbef193db87326132
I20260928 17:47:14.252884 12180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:14.261564 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:14.242970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.3.222.62:46647)
0928 17:47:14.243173 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:14.243184 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:14.243232 (+ 48us) server_negotiation.cc:407] Connection header received
0928 17:47:14.243718 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:14.243727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:14.243788 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:14.243928 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:14.244517 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.245843 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.246736 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.247249 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.247366 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:14.257639 (+ 10273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:14.257644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:14.257644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:14.257657 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:14.259572 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:14.259958 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:14.259962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:14.259962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:14.260070 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:14.260297 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:14.260298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:14.260298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:14.260601 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:14.260734 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:14.261193 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 17:47:14.261344 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260928 17:47:14.274349 12180 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:14.275071 12180 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:14.282909 12180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: Generated new TSK 0
I20260928 17:47:14.283092 12180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:47:14.314138 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:46647
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:14.395462 12194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:14.395654 12194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:14.395676 12194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:14.396767 12194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:14.396799 12194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:14.396822 12194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:47:14.398186 12194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:14.398377 12194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:14.398543 12194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:14.403247 12199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:14.403261 12202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:14.403352 12200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:14.403381 12194 server_base.cc:1034] running on GCE node
I20260928 17:47:14.403641 12194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:14.403846 12194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:14.404979 12194 hybrid_clock.cc:648] HybridClock initialized: now 1790617634404963 us; error 27 us; skew 500 ppm
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:14.407401 12194 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:47:14.407792 12194 webserver.cc:460] Webserver started at http://127.3.222.1:35093/ using document root <none> and password file <none>
I20260928 17:47:14.407997 12194 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:14.408042 12194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:14.408154 12194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:14.409111 12194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "155fcf99363b4341b16b81517ef99d93"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "85e5de1d0adde33e01535d5b0c54cc0b"
server_key_iv: "c2abcd876622cb860f5de7818eb18e71"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.409415 12194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "155fcf99363b4341b16b81517ef99d93"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "85e5de1d0adde33e01535d5b0c54cc0b"
server_key_iv: "c2abcd876622cb860f5de7818eb18e71"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.410537 12194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:47:14.411116 12209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.411285 12194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:14.411361 12194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "155fcf99363b4341b16b81517ef99d93"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "85e5de1d0adde33e01535d5b0c54cc0b"
server_key_iv: "c2abcd876622cb860f5de7818eb18e71"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.411419 12194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:14.419741 12194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:14.421592 12194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:14.421713 12194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:14.421890 12194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:14.422144 12194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:14.422174 12194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.422201 12194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:14.422224 12194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.427584 12194 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:43245
I20260928 17:47:14.427640 12322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:43245 every 8 connection(s)
I20260928 17:47:14.427903 12194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:47:14.428486 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 12194
I20260928 17:47:14.428586 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:47:14.428803 3960 external_mini_cluster.cc:1428] Setting key afcff43720f7c9142b797771267ee621
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
I20260928 17:47:14.436885 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:14.428535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:35111 (local address 127.3.222.62:46647)
0928 17:47:14.428778 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:14.428780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:14.429108 (+ 328us) server_negotiation.cc:407] Connection header received
0928 17:47:14.429674 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:14.429675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:14.429686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:14.429730 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:14.430622 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.431113 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.432116 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.432377 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.432440 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:14.434475 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:14.434477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:14.434478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:14.434485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:14.435705 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:14.436048 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:14.436049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:14.436049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:14.436097 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:14.436309 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:14.436310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:14.436310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:14.436413 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:14.436467 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:14.436714 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 17:47:14.436794 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260928 17:47:14.437034 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:14.428536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:35111)
0928 17:47:14.428985 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:14.429020 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:14.429575 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:14.429820 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:14.429824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:14.430345 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:14.430526 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:14.430532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.431198 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.431199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:14.432030 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:14.432032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.432451 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.432452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:14.432565 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:14.432996 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:14.433004 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:14.434393 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:14.435776 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:14.435782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:14.435785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:14.435985 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:14.436151 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:14.436159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:14.436159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:14.436254 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:14.436467 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:14.436473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:14.436601 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 17:47:14.436687 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 17:47:14.436831 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":79,"threads_started":1}
I20260928 17:47:14.437484 12323 heartbeater.cc:347] Connected to a master server at 127.3.222.62:46647
I20260928 17:47:14.437565 12323 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:14.437724 12323 heartbeater.cc:511] Master 127.3.222.62:46647 requested a full tablet report, sending...
I20260928 17:47:14.438174 12132 ts_manager.cc:194] Registered new tserver with Master: 155fcf99363b4341b16b81517ef99d93 (127.3.222.1:43245)
I20260928 17:47:14.438836 12132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:35111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:47:14.453759 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:46647
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:14.531373 12328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:14.531572 12328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:14.531601 12328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:14.532742 12328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:14.532778 12328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:14.532812 12328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:47:14.534178 12328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:14.534399 12328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:14.534597 12328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:14.539105 12336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:14.539161 12334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:14.539084 12333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:14.539253 12328 server_base.cc:1034] running on GCE node
I20260928 17:47:14.539435 12328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:14.539641 12328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:14.540777 12328 hybrid_clock.cc:648] HybridClock initialized: now 1790617634540766 us; error 26 us; skew 500 ppm
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:14.542958 12328 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:47:14.543359 12328 webserver.cc:460] Webserver started at http://127.3.222.2:44921/ using document root <none> and password file <none>
I20260928 17:47:14.543527 12328 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:14.543562 12328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:14.543648 12328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:14.544579 12328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "df0442748a214baf93691787bb02d717"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "18e7f0b7cfc0a623110213d509b86902"
server_key_iv: "cad07889174da709f4f5bbf1221acbd2"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.544873 12328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "df0442748a214baf93691787bb02d717"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "18e7f0b7cfc0a623110213d509b86902"
server_key_iv: "cad07889174da709f4f5bbf1221acbd2"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.545993 12328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:47:14.546582 12343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.546725 12328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:14.546795 12328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "df0442748a214baf93691787bb02d717"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "18e7f0b7cfc0a623110213d509b86902"
server_key_iv: "cad07889174da709f4f5bbf1221acbd2"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.546857 12328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:14.575169 12328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:14.576929 12328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:14.577049 12328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:14.577263 12328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:14.577550 12328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:14.577581 12328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.577603 12328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:14.577626 12328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.583092 12328 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:41063
I20260928 17:47:14.583149 12456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:41063 every 8 connection(s)
I20260928 17:47:14.583452 12328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:47:14.588064 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 12328
I20260928 17:47:14.588150 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:47:14.588308 3960 external_mini_cluster.cc:1428] Setting key 32cdda9de5ea8c093b2839ff23924328
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:14.592324 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:14.584110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:51209 (local address 127.3.222.62:46647)
0928 17:47:14.584163 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:14.584165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:14.584369 (+ 204us) server_negotiation.cc:407] Connection header received
0928 17:47:14.584863 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:14.584864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:14.584874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:14.584934 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:14.585818 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.586301 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.587246 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.587464 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.587515 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:14.589756 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:14.589757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:14.589758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:14.589764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:14.591000 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:14.591410 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:14.591411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:14.591412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:14.591472 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:14.591681 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:14.591682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:14.591682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:14.591799 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:14.591857 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:14.592199 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 17:47:14.592259 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:47:14.592551 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:14.584077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:51209)
0928 17:47:14.584221 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:14.584262 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:14.584750 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:14.585006 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:14.585009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:14.585540 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:14.585727 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:14.585731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.586358 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.586359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:14.587150 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:14.587152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.587521 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.587522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:14.587611 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:14.588084 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:14.588091 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:14.589652 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:14.591073 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:14.591080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:14.591082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:14.591346 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:14.591525 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:14.591526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:14.591526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:14.591627 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:14.591880 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:14.591886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:14.592051 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 17:47:14.592167 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 17:47:14.592339 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260928 17:47:14.592969 12457 heartbeater.cc:347] Connected to a master server at 127.3.222.62:46647
I20260928 17:47:14.593047 12457 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:14.593209 12457 heartbeater.cc:511] Master 127.3.222.62:46647 requested a full tablet report, sending...
I20260928 17:47:14.593557 12132 ts_manager.cc:194] Registered new tserver with Master: df0442748a214baf93691787bb02d717 (127.3.222.2:41063)
I20260928 17:47:14.593904 12132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:51209
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:47:14.614975 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:46647
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:14.692428 12462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:14.692633 12462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:14.692664 12462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:14.693822 12462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:14.693861 12462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:14.693897 12462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:14.695353 12462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:14.695570 12462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:14.695778 12462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:14.700312 12468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:14.700337 12470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:14.700380 12467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:14.700402 12462 server_base.cc:1034] running on GCE node
I20260928 17:47:14.700661 12462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:14.700848 12462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:14.701972 12462 hybrid_clock.cc:648] HybridClock initialized: now 1790617634701962 us; error 27 us; skew 500 ppm
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:14.704231 12462 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:47:14.704607 12462 webserver.cc:460] Webserver started at http://127.3.222.3:33999/ using document root <none> and password file <none>
I20260928 17:47:14.704831 12462 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:14.704875 12462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:14.704988 12462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:14.705996 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "e920ef76308b4f449b759dfa29837a17"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "08b53a5931174eb53770afd2eed38d24"
server_key_iv: "9ddd9e5bc2250b5fa82b3a5c9f90589e"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.706308 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "e920ef76308b4f449b759dfa29837a17"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "08b53a5931174eb53770afd2eed38d24"
server_key_iv: "9ddd9e5bc2250b5fa82b3a5c9f90589e"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.707516 12462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:47:14.708245 12477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.708389 12462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:14.708465 12462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "e920ef76308b4f449b759dfa29837a17"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "08b53a5931174eb53770afd2eed38d24"
server_key_iv: "9ddd9e5bc2250b5fa82b3a5c9f90589e"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.708527 12462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:14.725874 12462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:14.727792 12462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:14.727907 12462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:14.728120 12462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:14.728405 12462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:14.728441 12462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.728463 12462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:14.728487 12462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.735256 12462 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:37649
I20260928 17:47:14.735359 12590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:37649 every 8 connection(s)
I20260928 17:47:14.735682 12462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:47:14.739316 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 12462
I20260928 17:47:14.739411 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:47:14.739569 3960 external_mini_cluster.cc:1428] Setting key 229f10731b3d649f1d5a85f8c4f9a70e
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:14.744017 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:14.736360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:34697 (local address 127.3.222.62:46647)
0928 17:47:14.736400 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:14.736402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:14.736627 (+ 225us) server_negotiation.cc:407] Connection header received
0928 17:47:14.737099 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:14.737101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:14.737113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:14.737164 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:14.738068 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.738568 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.739614 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.739854 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.739916 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:14.741899 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:14.741900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:14.741900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:14.741906 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:14.742956 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:14.743327 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:14.743328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:14.743328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:14.743372 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:14.743542 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:14.743543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:14.743543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:14.743633 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:14.743701 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:14.743902 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 17:47:14.743958 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:47:14.744200 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:14.736352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:34697)
0928 17:47:14.736512 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:14.736548 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:14.737009 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:14.737240 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:14.737244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:14.737796 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:14.737979 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:14.737984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.738647 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.738648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:14.739542 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:14.739545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:14.739936 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:14.739938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:14.740097 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:14.740588 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:14.740596 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:14.741823 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:14.743016 (+ 1193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:14.743022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:14.743024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:14.743268 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:14.743415 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:14.743416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:14.743416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:14.743502 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:14.743696 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:14.743700 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:14.743808 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 17:47:14.743882 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 17:47:14.743994 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260928 17:47:14.744616 12591 heartbeater.cc:347] Connected to a master server at 127.3.222.62:46647
I20260928 17:47:14.744700 12591 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:14.744859 12591 heartbeater.cc:511] Master 127.3.222.62:46647 requested a full tablet report, sending...
I20260928 17:47:14.745198 12132 ts_manager.cc:194] Registered new tserver with Master: e920ef76308b4f449b759dfa29837a17 (127.3.222.3:37649)
I20260928 17:47:14.745523 12132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:34697
I20260928 17:47:14.751132 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab.
Entry for principal oryx/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab.
WARNING: no policy specified for oryx/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab.
Entry for principal oryx/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab.
WARNING: no policy specified for oryx/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab.
Entry for principal oryx/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab.
I20260928 17:47:14.817903 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 12194
I20260928 17:47:14.822157 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 12328
I20260928 17:47:14.825562 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 12462
I20260928 17:47:14.828910 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 12100
I20260928 17:47:14.832809 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:46647
--webserver_interface=127.3.222.62
--webserver_port=46059
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:46647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:14.913287 12600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:14.913475 12600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:14.913496 12600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:14.916172 12600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:14.916230 12600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:14.916245 12600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:14.916256 12600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:14.916280 12600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:14.918318 12600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:46647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:46647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=46059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:14.918553 12600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:14.918761 12600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:14.923166 12606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:14.923158 12605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:14.923385 12600 server_base.cc:1034] running on GCE node
W20260928 17:47:14.923398 12608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:14.923666 12600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:14.923942 12600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:14.925091 12600 hybrid_clock.cc:648] HybridClock initialized: now 1790617634925077 us; error 31 us; skew 500 ppm
Sep 28 17:47:14 dist-test-slave-w3bm krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:14.927464 12600 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:47:14.927872 12600 webserver.cc:460] Webserver started at http://127.3.222.62:46059/ using document root <none> and password file <none>
I20260928 17:47:14.928083 12600 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:14.928135 12600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:14.929446 12600 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:14.930060 12615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:14.930244 12600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:14.930325 12600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "69f941d8b9124a3e9f0262e90b74a31a"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "4fa8414ff3c5263634dfad7f951a9e23"
server_key_iv: "754d751cb76ee23878a46f3c545b490e"
server_key_version: "encryptionkey@0"
I20260928 17:47:14.930697 12600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:14.943295 12600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:14.945155 12600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:14.945283 12600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:14.949195 12600 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:46647
I20260928 17:47:14.949281 12667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:46647 every 8 connection(s)
I20260928 17:47:14.949604 12600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:14.951042 12668 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:47:14.951681 12668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: Bootstrap starting.
I20260928 17:47:14.956836 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 12600
I20260928 17:47:14.957172 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:43245
--local_ip_for_outbound_sockets=127.3.222.1
--tserver_master_addrs=127.3.222.62:46647
--webserver_port=35093
--webserver_interface=127.3.222.1
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.222.1
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:47:14.960671 12668 log.cc:824] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:14.962373 12668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:47:14.962621 12668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: Bootstrap complete.
I20260928 17:47:14.964691 12668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } }
I20260928 17:47:14.964897 12668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f941d8b9124a3e9f0262e90b74a31a, State: Initialized, Role: FOLLOWER
I20260928 17:47:14.965045 12668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } }
I20260928 17:47:14.965118 12668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:14.965166 12668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:14.965209 12668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:47:14.966131 12668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } }
I20260928 17:47:14.966219 12668 leader_election.cc:302] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f941d8b9124a3e9f0262e90b74a31a; no voters:
I20260928 17:47:14.966362 12668 leader_election.cc:288] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 17:47:14.966403 12672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:47:14.966514 12672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [term 2 LEADER]: Becoming Leader. State: Replica: 69f941d8b9124a3e9f0262e90b74a31a, State: Running, Role: LEADER
I20260928 17:47:14.966593 12672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } }
I20260928 17:47:14.966620 12668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:14.966868 12674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f941d8b9124a3e9f0262e90b74a31a. Latest consensus state: current_term: 2 leader_uuid: "69f941d8b9124a3e9f0262e90b74a31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } } }
I20260928 17:47:14.966878 12673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "69f941d8b9124a3e9f0262e90b74a31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f941d8b9124a3e9f0262e90b74a31a" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46647 } } }
I20260928 17:47:14.966956 12674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:14.966961 12673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:14.967161 12676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:14.967648 12676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:14.967780 12676 catalog_manager.cc:1259] Loaded cluster ID: 67f2d76c9f8d47ebbef193db87326132
I20260928 17:47:14.967818 12676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:47:14.968649 12676 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:14.968799 12676 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 69f941d8b9124a3e9f0262e90b74a31a: Loaded TSK: 0
I20260928 17:47:14.968976 12676 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 17:47:15.035674 12670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:15.035863 12670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:15.035893 12670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:15.037021 12670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:15.037060 12670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:15.037096 12670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:47:15.038476 12670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.1.keytab
--principal=oryx/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:43245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=35093
--tserver_master_addrs=127.3.222.62:46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:15.038686 12670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:15.038888 12670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:15.043649 12695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:15.043629 12694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:15.043622 12697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:15.044153 12670 server_base.cc:1034] running on GCE node
I20260928 17:47:15.044382 12670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:15.044610 12670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:15.045743 12670 hybrid_clock.cc:648] HybridClock initialized: now 1790617635045732 us; error 30 us; skew 500 ppm
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617635, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:15.048249 12670 init.cc:377] Logged in from keytab as oryx/127.3.222.1@KRBTEST.COM (short username oryx)
I20260928 17:47:15.048713 12670 webserver.cc:460] Webserver started at http://127.3.222.1:35093/ using document root <none> and password file <none>
I20260928 17:47:15.048926 12670 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:15.048986 12670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:15.050267 12670 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:15.050818 12704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.050946 12670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:15.051025 12670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "155fcf99363b4341b16b81517ef99d93"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "85e5de1d0adde33e01535d5b0c54cc0b"
server_key_iv: "c2abcd876622cb860f5de7818eb18e71"
server_key_version: "encryptionkey@0"
I20260928 17:47:15.051405 12670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:15.064016 12670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:15.065914 12670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:15.066021 12670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:15.066218 12670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:15.066519 12670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:15.066552 12670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.066572 12670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:15.066586 12670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.073149 12670 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:43245
I20260928 17:47:15.073203 12817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:43245 every 8 connection(s)
I20260928 17:47:15.073498 12670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:15.081987 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 12670
I20260928 17:47:15.082257 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:41063
--local_ip_for_outbound_sockets=127.3.222.2
--tserver_master_addrs=127.3.222.62:46647
--webserver_port=44921
--webserver_interface=127.3.222.2
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.222.2
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:47:15.082463 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.074133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:35239)
0928 17:47:15.074310 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:15.074355 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:15.074967 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:15.075395 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:15.075399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:15.076118 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:15.076371 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.076378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.077529 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.077530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.078730 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.078733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.079354 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.079355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.079493 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.079993 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:15.080002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:15.082261 (+ 2259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
W20260928 17:47:15.082674 12818 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:46647 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.222.62:46647: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
W20260928 17:47:15.083122 12821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:15.074244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:35239 (local address 127.3.222.62:46647)
0928 17:47:15.074397 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.074406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.074439 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:47:15.075152 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.075157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.075197 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.075337 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.076469 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.077437 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.078821 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.079280 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.079359 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.082929 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:35239: BlockingRecv error: recv got EOF from 127.3.222.1:35239 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:15.088320 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.082858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:50237)
0928 17:47:15.083108 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:15.083118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:15.083158 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:15.083533 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:15.083535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:15.083660 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:15.083710 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.083711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.084615 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.084616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.085334 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.085337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.085820 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.085821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.085887 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.086321 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:15.086326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:15.088247 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":62,"threads_started":1}
W20260928 17:47:15.088421 12823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:15.083002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:50237 (local address 127.3.222.62:46647)
0928 17:47:15.083285 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.083287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.083295 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:15.083316 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.083316 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.083327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.083775 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.083807 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.084541 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.085415 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.085738 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.085810 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.088353 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:50237: BlockingRecv error: recv got EOF from 127.3.222.1:50237 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":55,"threads_started":1}
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:15.093719 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.088796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:56573)
0928 17:47:15.088873 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:15.088883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:15.088929 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:15.089167 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:15.089168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:15.089297 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:15.089329 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.089331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.090185 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.090186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.090943 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.090946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.091457 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.091459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.091522 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.091918 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:15.091922 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:15.093651 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 17:47:15.093957 12823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:15.088839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:56573 (local address 127.3.222.62:46647)
0928 17:47:15.088881 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.088883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.088917 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:47:15.088975 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.088976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.088986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.089127 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.089410 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.090098 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.091035 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.091346 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.091405 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.093898 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:56573: BlockingRecv error: recv got EOF from 127.3.222.1:56573 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 17:47:15.093943 12818 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:47:15.163065 12824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:15.163268 12824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:15.163298 12824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:15.164446 12824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:15.164486 12824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:15.164520 12824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:47:15.165886 12824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.2.keytab
--principal=oryx/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:41063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=44921
--tserver_master_addrs=127.3.222.62:46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:15.166092 12824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:15.166277 12824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:15.171147 12829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:15.171113 12830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:15.171116 12832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:15.171399 12824 server_base.cc:1034] running on GCE node
I20260928 17:47:15.171609 12824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:15.171800 12824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:15.172962 12824 hybrid_clock.cc:648] HybridClock initialized: now 1790617635172942 us; error 35 us; skew 500 ppm
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617635, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:15.175244 12824 init.cc:377] Logged in from keytab as oryx/127.3.222.2@KRBTEST.COM (short username oryx)
I20260928 17:47:15.175648 12824 webserver.cc:460] Webserver started at http://127.3.222.2:44921/ using document root <none> and password file <none>
I20260928 17:47:15.175863 12824 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:15.175922 12824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:15.177139 12824 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:15.177675 12839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.177826 12824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.177911 12824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "df0442748a214baf93691787bb02d717"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "18e7f0b7cfc0a623110213d509b86902"
server_key_iv: "cad07889174da709f4f5bbf1221acbd2"
server_key_version: "encryptionkey@0"
I20260928 17:47:15.178218 12824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:15.189911 12824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:15.191828 12824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:15.191946 12824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:15.192135 12824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:15.192389 12824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:15.192420 12824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.192445 12824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:15.192469 12824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.197939 12824 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:41063
I20260928 17:47:15.197980 12952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:41063 every 8 connection(s)
I20260928 17:47:15.198272 12824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 17:47:15.204744 12823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:15.198903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:48809 (local address 127.3.222.62:46647)
0928 17:47:15.198953 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.198955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.199138 (+ 183us) server_negotiation.cc:407] Connection header received
0928 17:47:15.199623 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.199624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.199641 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.199699 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.200561 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.201051 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.201971 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.202160 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.202213 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.204699 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:48809: BlockingRecv error: recv got EOF from 127.3.222.2:48809 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:47:15.204748 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.198870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:48809)
0928 17:47:15.199005 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:15.199043 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:15.199542 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:15.199749 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:15.199752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:15.200271 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:15.200474 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.200479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.201130 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.201131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.201899 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.201901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.202219 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.202220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.202312 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.202730 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:15.202738 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:15.204618 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
W20260928 17:47:15.204985 12953 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:46647 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.222.62:46647: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
I20260928 17:47:15.207145 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 12824
I20260928 17:47:15.207448 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:37649
--local_ip_for_outbound_sockets=127.3.222.3
--tserver_master_addrs=127.3.222.62:46647
--webserver_port=33999
--webserver_interface=127.3.222.3
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.222.3
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:15.209180 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.205183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:57833)
0928 17:47:15.205232 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:15.205243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:15.205284 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:15.205419 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:15.205419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:15.205523 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:15.205561 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.205562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.206109 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.206109 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.206610 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.206611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.206943 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.206944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.206994 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.207390 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:15.207393 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:15.209122 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 17:47:15.209252 12823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:15.205236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:57833 (local address 127.3.222.62:46647)
0928 17:47:15.205306 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.205308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.205314 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:47:15.205332 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.205332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.205341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.205383 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.205616 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.206058 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.206690 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.206875 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.206934 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.209202 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:57833: BlockingRecv error: recv got EOF from 127.3.222.2:57833 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:15.213191 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.209541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:53843)
0928 17:47:15.209579 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:15.209588 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:15.209623 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:15.209720 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:15.209721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:15.209840 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:15.209863 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.209865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.210423 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.210424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.210912 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.210914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.211223 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.211225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.211273 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.211599 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:15.211603 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:15.213135 (+ 1532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 17:47:15.213296 12823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:15.209589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:53843 (local address 127.3.222.62:46647)
0928 17:47:15.209627 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.209635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.209643 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:15.209670 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.209671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.209679 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.209718 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.209920 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.210361 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.210976 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.211152 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.211200 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.213250 (+ 2050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:53843: BlockingRecv error: recv got EOF from 127.3.222.2:53843 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:47:15.213450 12953 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:47:15.284909 12956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:15.285094 12956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:15.285121 12956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:15.286222 12956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:15.286259 12956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:15.286302 12956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:15.287703 12956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/krb5kdc/oryx_127.3.222.3.keytab
--principal=oryx/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:37649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=33999
--tserver_master_addrs=127.3.222.62:46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:15.287918 12956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:15.288103 12956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:15.292758 12962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:15.292701 12964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:15.292781 12961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:15.292886 12956 server_base.cc:1034] running on GCE node
I20260928 17:47:15.293126 12956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:15.293344 12956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:15.294484 12956 hybrid_clock.cc:648] HybridClock initialized: now 1790617635294474 us; error 30 us; skew 500 ppm
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617635, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:15.296804 12956 init.cc:377] Logged in from keytab as oryx/127.3.222.3@KRBTEST.COM (short username oryx)
I20260928 17:47:15.297200 12956 webserver.cc:460] Webserver started at http://127.3.222.3:33999/ using document root <none> and password file <none>
I20260928 17:47:15.297415 12956 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:15.297473 12956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:15.298714 12956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:15.299263 12971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.299401 12956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:15.299480 12956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "e920ef76308b4f449b759dfa29837a17"
format_stamp: "Formatted at 2026-09-28 17:47:14 on dist-test-slave-w3bm"
server_key: "08b53a5931174eb53770afd2eed38d24"
server_key_iv: "9ddd9e5bc2250b5fa82b3a5c9f90589e"
server_key_version: "encryptionkey@0"
I20260928 17:47:15.299866 12956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:15.317761 12956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:15.319749 12956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:15.319875 12956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:15.320083 12956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:15.320390 12956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:15.320422 12956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.320448 12956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:15.320472 12956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:15.325793 12956 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:37649
I20260928 17:47:15.325843 13084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:37649 every 8 connection(s)
I20260928 17:47:15.326133 12956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:15.332083 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 12956
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:15.333032 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.326759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:36507)
0928 17:47:15.326909 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:15.326943 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:15.327450 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:15.327667 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:15.327671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:15.328189 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:15.328385 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.328389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.329059 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.329060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.329836 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.329838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.330179 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.330179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.330296 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.330708 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:15.330716 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:15.332896 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260928 17:47:15.333032 12823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:15.326770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:36507 (local address 127.3.222.62:46647)
0928 17:47:15.326825 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.326827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.327017 (+ 190us) server_negotiation.cc:407] Connection header received
0928 17:47:15.327538 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.327540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.327551 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.327601 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.328472 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.328978 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.329914 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.330114 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.330168 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.332973 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:36507: BlockingRecv error: recv got EOF from 127.3.222.3:36507 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 17:47:15.333271 13085 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:46647 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.222.62:46647: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:15.337203 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.333457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:45301)
0928 17:47:15.333511 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:15.333520 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:15.333555 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:15.333792 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:15.333793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:15.333915 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:15.333954 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.333955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.334520 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.334521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.335007 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.335009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.335345 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.335346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.335415 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.335754 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:15.335758 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:15.337141 (+ 1383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 17:47:15.337271 12823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:15.333509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:45301 (local address 127.3.222.62:46647)
0928 17:47:15.333709 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.333710 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.333716 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:47:15.333731 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.333731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.333738 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.333780 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.334016 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.334459 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.335073 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.335293 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.335345 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.337221 (+ 1876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:45301: BlockingRecv error: recv got EOF from 127.3.222.3:45301 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
I20260928 17:47:15.338910 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.332571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.3.222.62:46647)
0928 17:47:15.332614 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.332615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.332662 (+ 47us) server_negotiation.cc:407] Connection header received
0928 17:47:15.332734 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.332735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.332748 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.332792 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.333442 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.333975 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.334746 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.334932 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.334981 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.336149 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:15.336153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:15.336154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:15.336164 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:15.337741 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:15.338047 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:15.338050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:15.338051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:15.338132 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:15.338296 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:15.338297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:15.338297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:15.338508 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:15.338632 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:15.338735 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 17:47:15.338836 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:15 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:15.340984 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:15.337433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:60385)
0928 17:47:15.337513 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:15.337522 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:15.337562 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:15.337661 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:15.337662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:15.337767 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:15.337790 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.337791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.338336 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.338337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.338801 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:15.338803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.339167 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.339168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:15.339243 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.339579 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:15.339583 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:15.340944 (+ 1361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 17:47:15.341085 12823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:15.337481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:60385 (local address 127.3.222.62:46647)
0928 17:47:15.337513 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:15.337514 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:15.337550 (+ 36us) server_negotiation.cc:407] Connection header received
0928 17:47:15.337610 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:15.337610 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:15.337618 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:15.337660 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:15.337848 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.338271 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.338865 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:15.339096 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:15.339155 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:15.341028 (+ 1873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:60385: BlockingRecv error: recv got EOF from 127.3.222.3:60385 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 17:47:15.341203 13085 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 17:47:16 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:16 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:16.098410 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:16.094285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:44735)
0928 17:47:16.094399 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:16.094409 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:16.094453 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:16.094582 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:16.094583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:16.094727 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:16.094782 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:16.094783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.095450 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.095451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:16.096001 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:16.096003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.096349 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.096350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:16.096388 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:16.096796 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:16.096800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:16.098346 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":50,"threads_started":1}
W20260928 17:47:16.098510 13090 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:16.094367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:44735 (local address 127.3.222.62:46647)
0928 17:47:16.094477 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:16.094479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:16.094486 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:16.094506 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:16.094506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:16.094514 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:16.094573 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:16.094862 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.095390 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.096083 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.096286 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.096345 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:16.098434 (+ 2089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:44735: BlockingRecv error: recv got EOF from 127.3.222.1:44735 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
Sep 28 17:47:16 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:16 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:16.218225 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:16.213839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:49407)
0928 17:47:16.213963 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:16.213973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:16.214023 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:16.214136 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:16.214137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:16.214293 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:16.214343 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:16.214345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.214991 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.214992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:16.215645 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:16.215648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.216006 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.216007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:16.216044 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:16.216467 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:16.216472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:16.218159 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
W20260928 17:47:16.218325 13090 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:16.213907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:49407 (local address 127.3.222.62:46647)
0928 17:47:16.213987 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:16.213989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:16.214012 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:47:16.214064 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:16.214065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:16.214075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:16.214126 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:16.214404 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.214889 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.215717 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.215920 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.215980 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:16.218265 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:49407: BlockingRecv error: recv got EOF from 127.3.222.2:49407 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 17:47:16 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:16 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:16.346030 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:16.341554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:34271)
0928 17:47:16.341689 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:16.341699 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:16.341751 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:16.341880 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:16.341881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:16.342042 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:16.342090 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:16.342091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.342771 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.342772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:16.343383 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:16.343386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.343720 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.343721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:16.343776 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:16.344225 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:16.344230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:16.345961 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
W20260928 17:47:16.346141 13090 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:16.341629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:34271 (local address 127.3.222.62:46647)
0928 17:47:16.341699 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:16.341701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:16.341742 (+ 41us) server_negotiation.cc:407] Connection header received
0928 17:47:16.341791 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:16.341792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:16.341804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:16.341856 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:16.342185 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.342693 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.343464 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:16.343653 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:16.343706 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:16.346070 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:34271: BlockingRecv error: recv got EOF from 127.3.222.3:34271 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 17:47:17 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:17 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:17.103293 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:17.098831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:50687)
0928 17:47:17.098959 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:17.098968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:17.099012 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:17.099149 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:17.099150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:17.099342 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:17.099392 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:17.099394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.100024 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.100025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:17.100555 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:17.100557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.101036 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.101037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:17.101085 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:17.101561 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:17.101565 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:17.103189 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260928 17:47:17.103401 13094 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:17.098891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:50687 (local address 127.3.222.62:46647)
0928 17:47:17.099011 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:17.099013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:17.099021 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:17.099068 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:17.099069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:17.099078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:17.099141 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:17.099451 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.099951 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.100682 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.100953 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.101019 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:17.103315 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:50687: BlockingRecv error: recv got EOF from 127.3.222.1:50687 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
Sep 28 17:47:17 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:17 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:17.223258 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:17.218656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:56643)
0928 17:47:17.218815 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:17.218835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:17.218895 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:17.219014 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:17.219014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:17.219168 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:17.219246 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:17.219248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.219880 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.219881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:17.220447 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:17.220450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.220850 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.220852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:17.220904 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:17.221328 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:17.221333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:17.223169 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
W20260928 17:47:17.223440 13094 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:17.218739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:56643 (local address 127.3.222.62:46647)
0928 17:47:17.218801 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:17.218803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:17.218880 (+ 77us) server_negotiation.cc:407] Connection header received
0928 17:47:17.218944 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:17.218945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:17.218955 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:17.219000 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:17.219302 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.219800 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.220572 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.220765 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.220825 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:17.223336 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:56643: BlockingRecv error: recv got EOF from 127.3.222.2:56643 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 17:47:17 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:17 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:17.350730 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:17.346473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:59725)
0928 17:47:17.346608 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:17.346619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:17.346670 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:17.346797 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:17.346798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:17.346963 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:17.347008 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:17.347010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.347689 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.347690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:17.348233 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:17.348235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.348581 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.348582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:17.348638 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:17.349066 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:17.349071 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:17.350654 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
W20260928 17:47:17.350800 13094 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:17.346614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:59725 (local address 127.3.222.62:46647)
0928 17:47:17.346665 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:17.346666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:17.346674 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:17.346716 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:17.346717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:17.346731 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:17.346783 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:17.347076 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.347611 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.348340 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:17.348523 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:17.348576 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:17.350747 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:59725: BlockingRecv error: recv got EOF from 127.3.222.3:59725 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 17:47:18 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:18 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 17:47:18.108353 13098 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:18.103815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:58991 (local address 127.3.222.62:46647)
0928 17:47:18.103915 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:18.103917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:18.103961 (+ 44us) server_negotiation.cc:407] Connection header received
0928 17:47:18.104002 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:18.104002 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:18.104012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:18.104069 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:18.104439 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.105068 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.105814 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.106012 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.106070 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:18.108297 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:58991: BlockingRecv error: recv got EOF from 127.3.222.1:58991 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
I20260928 17:47:18.108368 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:18.103723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:58991)
0928 17:47:18.103872 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:18.103884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:18.103957 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:18.104117 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:18.104118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:18.104301 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:18.104362 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:18.104364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.105158 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.105160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:18.105727 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:18.105729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.106074 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.106074 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:18.106114 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:18.106506 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:18.106510 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:18.108201 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
Sep 28 17:47:18 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:18 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:18.227993 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:18.223682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:45233)
0928 17:47:18.223831 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:18.223843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:18.223912 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:18.224038 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:18.224039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:18.224226 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:18.224273 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:18.224275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.224910 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.224911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:18.225452 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:18.225454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.225788 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.225789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:18.225844 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:18.226241 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:18.226246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:18.227908 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
W20260928 17:47:18.228066 13098 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:18.223751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:45233 (local address 127.3.222.62:46647)
0928 17:47:18.223828 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:18.223830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:18.223892 (+ 62us) server_negotiation.cc:407] Connection header received
0928 17:47:18.223965 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:18.223966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:18.223977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:18.224023 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:18.224345 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.224840 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.225528 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.225728 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.225781 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:18.228013 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:45233: BlockingRecv error: recv got EOF from 127.3.222.2:45233 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 17:47:18 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:18 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:18.355575 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:18.351243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:54417)
0928 17:47:18.351366 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:18.351377 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:18.351433 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:18.351561 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:18.351562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:18.351710 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:18.351758 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:18.351759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.352418 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.352419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:18.352985 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:18.352987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.353292 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.353293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:18.353333 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:18.353787 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:18.353792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:18.355507 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
W20260928 17:47:18.355665 13098 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:18.351302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:54417 (local address 127.3.222.62:46647)
0928 17:47:18.351354 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:18.351356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:18.351420 (+ 64us) server_negotiation.cc:407] Connection header received
0928 17:47:18.351487 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:18.351488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:18.351499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:18.351550 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:18.351848 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.352336 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.353055 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:18.353234 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:18.353281 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:18.355600 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:54417: BlockingRecv error: recv got EOF from 127.3.222.3:54417 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 17:47:19 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:19 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:19.113013 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:19.108797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:55089)
0928 17:47:19.108949 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:19.108959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:19.109018 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:19.109184 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:19.109185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:19.109342 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:19.109392 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:19.109394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.110004 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.110004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:19.110554 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:19.110556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.110852 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.110852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:19.110887 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:19.111336 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:19.111340 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:19.112956 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
W20260928 17:47:19.113127 13102 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:19.108847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:55089 (local address 127.3.222.62:46647)
0928 17:47:19.108954 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:19.108956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:19.109019 (+ 63us) server_negotiation.cc:407] Connection header received
0928 17:47:19.109068 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:19.109069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:19.109078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:19.109128 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:19.109452 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.109938 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.110619 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.110802 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.110846 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:19.113048 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:55089: BlockingRecv error: recv got EOF from 127.3.222.1:55089 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
Sep 28 17:47:19 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:19 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:19.232555 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:19.228442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:45377)
0928 17:47:19.228597 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:19.228610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:19.228671 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:19.228784 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:19.228785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:19.228956 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:19.229006 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:19.229008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.229600 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.229601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:19.230123 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:19.230124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.230474 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.230475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:19.230526 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:19.230923 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:19.230927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:19.232498 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
W20260928 17:47:19.232642 13102 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:19.228535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:45377 (local address 127.3.222.62:46647)
0928 17:47:19.228603 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:19.228604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:19.228655 (+ 51us) server_negotiation.cc:407] Connection header received
0928 17:47:19.228721 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:19.228722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:19.228731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:19.228770 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:19.229056 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.229545 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.230197 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.230389 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.230458 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:19.232586 (+ 2128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:45377: BlockingRecv error: recv got EOF from 127.3.222.2:45377 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 17:47:19 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:19 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:19.363071 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:19.355993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:49625)
0928 17:47:19.356107 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:19.356118 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:19.356178 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:19.356277 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:19.356278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:19.356428 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:19.356476 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:19.356477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.357077 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.357077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:19.357606 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:19.357608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.357963 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.357964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:19.358006 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:19.361338 (+ 3332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:19.361344 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:19.362995 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
W20260928 17:47:19.363160 13102 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:19.356057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:49625 (local address 127.3.222.62:46647)
0928 17:47:19.356120 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:19.356122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:19.356162 (+ 40us) server_negotiation.cc:407] Connection header received
0928 17:47:19.356225 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:19.356226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:19.356237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:19.356273 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:19.356534 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.357023 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.357693 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:19.357887 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:19.357948 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:19.363096 (+ 5148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:49625: BlockingRecv error: recv got EOF from 127.3.222.3:49625 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 17:47:20 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:20 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:20.117781 13105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:20.113466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:48265)
0928 17:47:20.113603 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:20.113617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:20.113686 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:20.113813 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:20.113813 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:20.113987 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:20.114036 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:20.114037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.114664 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.114664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:20.115176 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:20.115178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.115548 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.115549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:20.115593 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:20.116068 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:20.116073 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:20.117706 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
W20260928 17:47:20.117861 13106 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:20.113582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:48265 (local address 127.3.222.62:46647)
0928 17:47:20.113701 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:20.113703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:20.113711 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:20.113732 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:20.113733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:20.113742 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:20.113794 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:20.114108 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.114607 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.115270 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.115486 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.115535 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:20.117800 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:48265: BlockingRecv error: recv got EOF from 127.3.222.1:48265 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
Sep 28 17:47:20 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:20 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:20.237557 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:20.232982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:47797)
0928 17:47:20.233117 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:20.233130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:20.233185 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:20.233298 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:20.233299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:20.233454 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:20.233498 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:20.233500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.234312 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.234313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:20.234877 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:20.234880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.235246 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.235247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:20.235304 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:20.235734 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:20.235739 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:20.237498 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
W20260928 17:47:20.237664 13106 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:20.233098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:47797 (local address 127.3.222.62:46647)
0928 17:47:20.233155 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:20.233157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:20.233169 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:47:20.233237 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:20.233238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:20.233248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:20.233292 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:20.233570 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.234228 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.234986 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.235169 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.235230 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:20.237584 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:47797: BlockingRecv error: recv got EOF from 127.3.222.2:47797 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 17:47:20 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:20 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:20.369138 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:20.363678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:43473)
0928 17:47:20.363848 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:20.363860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:20.363926 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:20.364129 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:20.364130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:20.364305 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:20.364445 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:20.364451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.365232 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.365233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:20.365922 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:20.365928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.366394 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.366395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:20.366461 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:20.366920 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:20.366925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:20.369055 (+ 2130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
W20260928 17:47:20.369258 13106 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:20.363800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:43473 (local address 127.3.222.62:46647)
0928 17:47:20.363876 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:20.363879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:20.363907 (+ 28us) server_negotiation.cc:407] Connection header received
0928 17:47:20.364031 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:20.364032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:20.364048 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:20.364114 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:20.364591 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.365148 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.366089 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:20.366321 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:20.366385 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:20.369170 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:43473: BlockingRecv error: recv got EOF from 127.3.222.3:43473 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 17:47:21 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:21 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:21.122375 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:21.118235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:55095)
0928 17:47:21.118363 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:21.118374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:21.118429 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:21.118550 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:21.118551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:21.118695 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:21.118743 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:21.118745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.119411 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.119412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:21.119944 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:21.119946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.120248 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.120249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:21.120287 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:21.120668 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:21.120672 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:21.122317 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
W20260928 17:47:21.122475 13110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:21.118317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:55095 (local address 127.3.222.62:46647)
0928 17:47:21.118421 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:21.118423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:21.118432 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:21.118475 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:21.118475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:21.118484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:21.118536 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:21.118797 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.119332 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.120015 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.120202 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.120243 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:21.122408 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:55095: BlockingRecv error: recv got EOF from 127.3.222.1:55095 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
Sep 28 17:47:21 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:21 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:21.242353 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:21.238006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:39269)
0928 17:47:21.238156 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:21.238168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:21.238246 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:21.238374 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:21.238375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:21.238549 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:21.238596 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:21.238598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.239255 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.239256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:21.239788 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:21.239790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.240123 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.240124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:21.240178 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:21.240587 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:21.240590 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:21.242278 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
W20260928 17:47:21.242441 13110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:21.238097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:39269 (local address 127.3.222.62:46647)
0928 17:47:21.238168 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:21.238170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:21.238241 (+ 71us) server_negotiation.cc:407] Connection header received
0928 17:47:21.238302 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:21.238303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:21.238314 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:21.238363 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:21.238665 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.239153 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.239870 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.240063 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.240119 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:21.242389 (+ 2270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:39269: BlockingRecv error: recv got EOF from 127.3.222.2:39269 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 17:47:21 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:21 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:21.374862 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:21.369829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:33911)
0928 17:47:21.369972 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:21.369982 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:21.370029 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:21.370199 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:21.370200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:21.370366 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:21.370422 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:21.370423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.371152 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.371153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:21.371913 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:21.371916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.372291 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.372292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:21.372347 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:21.372830 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:21.372836 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:21.374803 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
W20260928 17:47:21.374954 13110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:21.369967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:33911 (local address 127.3.222.62:46647)
0928 17:47:21.370019 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:21.370021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:21.370030 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:21.370080 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:21.370081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:21.370092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:21.370147 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:21.370496 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.371061 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.371990 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:21.372234 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:21.372290 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:21.374891 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:33911: BlockingRecv error: recv got EOF from 127.3.222.3:33911 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 17:47:22 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:22 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:22.127404 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:22.122844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:36767)
0928 17:47:22.122964 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:22.122977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:22.123045 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:22.123171 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:22.123172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:22.123342 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:22.123390 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:22.123392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.124038 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.124039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:22.124595 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:22.124597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.124926 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.124926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:22.124969 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:22.125447 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:22.125453 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:22.127346 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
W20260928 17:47:22.127504 13114 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:22.122921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:36767 (local address 127.3.222.62:46647)
0928 17:47:22.123064 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:22.123065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:22.123074 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:22.123095 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:22.123096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:22.123106 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:22.123161 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:22.123448 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.123947 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.124673 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.124867 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.124918 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:22.127436 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:36767: BlockingRecv error: recv got EOF from 127.3.222.1:36767 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
Sep 28 17:47:22 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:22 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:22.247403 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:22.242823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:47795)
0928 17:47:22.242955 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:22.242966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:22.243031 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:22.243262 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:22.243264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:22.243422 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:22.243469 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:22.243471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.244158 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.244159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:22.244701 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:22.244703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.245025 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.245026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:22.245067 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:22.245518 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:22.245523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:22.247329 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
W20260928 17:47:22.247473 13114 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:22.242904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:47795 (local address 127.3.222.62:46647)
0928 17:47:22.242979 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:22.242981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:22.243031 (+ 50us) server_negotiation.cc:407] Connection header received
0928 17:47:22.243075 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:22.243076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:22.243087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:22.243132 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:22.243566 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.244074 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.244776 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.244969 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.245023 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:22.247410 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:47795: BlockingRecv error: recv got EOF from 127.3.222.2:47795 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 17:47:22 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:22 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:22.380128 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:22.375305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:46363)
0928 17:47:22.375454 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:22.375465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:22.375521 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:22.375675 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:22.375676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:22.375857 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:22.375916 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:22.375918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.376805 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.376806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:22.377556 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:22.377559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.377976 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.377977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:22.378019 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:22.378410 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:22.378415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:22.380062 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
W20260928 17:47:22.380239 13114 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:22.375339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:46363 (local address 127.3.222.62:46647)
0928 17:47:22.375384 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:22.375386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:22.375510 (+ 124us) server_negotiation.cc:407] Connection header received
0928 17:47:22.375591 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:22.375592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:22.375607 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:22.375664 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:22.376017 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.376725 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.377705 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:22.377917 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:22.377978 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:22.380166 (+ 2188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:46363: BlockingRecv error: recv got EOF from 127.3.222.3:46363 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 17:47:23 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:23 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:23.132195 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:23.127860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:49763)
0928 17:47:23.127983 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:23.127993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:23.128037 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:23.128165 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:23.128165 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:23.128315 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:23.128364 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:23.128366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.128975 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.128976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:23.129494 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:23.129495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.129823 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.129824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:23.129868 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:23.130311 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:23.130316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:23.132120 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
W20260928 17:47:23.132377 13118 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:23.127948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:49763 (local address 127.3.222.62:46647)
0928 17:47:23.128061 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:23.128062 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:23.128069 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:47:23.128088 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:23.128089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:23.128098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:23.128154 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:23.128414 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.128920 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.129554 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.129761 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.129808 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:23.132222 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:49763: BlockingRecv error: recv got EOF from 127.3.222.1:49763 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
Sep 28 17:47:23 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:23 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:23.252427 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:23.247825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:53239)
0928 17:47:23.247965 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:23.247976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:23.248035 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:23.248209 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:23.248210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:23.248364 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:23.248413 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:23.248415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.249215 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.249216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:23.249747 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:23.249749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.250102 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.250108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:23.250174 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:23.250662 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:23.250667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:23.252350 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
W20260928 17:47:23.252514 13118 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:23.247881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:53239 (local address 127.3.222.62:46647)
0928 17:47:23.247933 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:23.247935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:23.248030 (+ 95us) server_negotiation.cc:407] Connection header received
0928 17:47:23.248080 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:23.248081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:23.248095 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:23.248170 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:23.248479 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.249144 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.249810 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.250038 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.250090 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:23.252460 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:53239: BlockingRecv error: recv got EOF from 127.3.222.2:53239 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 17:47:23 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:23 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:23.384685 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:23.380540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:46195)
0928 17:47:23.380667 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:23.380678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:23.380745 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:23.380916 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:23.380917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:23.381110 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:23.381170 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:23.381172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.381790 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.381791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:23.382341 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:23.382342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.382645 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.382645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:23.382681 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:23.383049 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:23.383053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:23.384627 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
W20260928 17:47:23.384773 13118 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:23.380585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:46195 (local address 127.3.222.62:46647)
0928 17:47:23.380644 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:23.380646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:23.380729 (+ 83us) server_negotiation.cc:407] Connection header received
0928 17:47:23.380830 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:23.380830 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:23.380845 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:23.380899 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:23.381231 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.381727 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.382408 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:23.382593 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:23.382641 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:23.384714 (+ 2073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:46195: BlockingRecv error: recv got EOF from 127.3.222.3:46195 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 17:47:24 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:24 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:24.136938 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:24.132673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:40545)
0928 17:47:24.132839 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:24.132852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:24.132909 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:24.133089 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:24.133090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:24.133257 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:24.133329 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:24.133331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.133998 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.133999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:24.134548 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:24.134550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.134872 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.134873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:24.134908 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:24.135339 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:24.135343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:24.136884 (+ 1541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
W20260928 17:47:24.137034 13122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:24.132771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:40545 (local address 127.3.222.62:46647)
0928 17:47:24.132889 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:24.132890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:24.132899 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:24.132958 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:24.132959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:24.132970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:24.133082 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:24.133414 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.133919 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.134620 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.134817 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.134868 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:24.136967 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:40545: BlockingRecv error: recv got EOF from 127.3.222.1:40545 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
Sep 28 17:47:24 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:24 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:24.257978 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:24.252937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:52341)
0928 17:47:24.253107 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:24.253119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:24.253196 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:24.253368 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:24.253368 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:24.253568 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:24.253627 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:24.253630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.254296 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.254297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:24.254879 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:24.254881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.255192 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.255193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:24.255255 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:24.255692 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:24.255697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:24.257886 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
W20260928 17:47:24.258296 13122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:24.253017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:52341 (local address 127.3.222.62:46647)
0928 17:47:24.253066 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:24.253068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:24.253187 (+ 119us) server_negotiation.cc:407] Connection header received
0928 17:47:24.253269 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:24.253269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:24.253281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:24.253346 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:24.253716 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.254211 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.254943 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.255129 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.255177 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:24.258236 (+ 3059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:52341: BlockingRecv error: recv got EOF from 127.3.222.2:52341 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 17:47:24 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:24 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:24.392674 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:24.385129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:48861)
0928 17:47:24.385245 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:24.385255 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:24.385310 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:24.385436 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:24.385436 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:24.385609 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:24.385656 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:24.385658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.386322 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.386323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:24.387098 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:24.387101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.387528 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.387529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:24.387571 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:24.390829 (+ 3258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:24.390834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:24.392587 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
W20260928 17:47:24.392743 13122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:24.385186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:48861 (local address 127.3.222.62:46647)
0928 17:47:24.385231 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:24.385233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:24.385299 (+ 66us) server_negotiation.cc:407] Connection header received
0928 17:47:24.385369 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:24.385369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:24.385382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:24.385428 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:24.385717 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.386245 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.387223 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:24.387458 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:24.387516 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:24.392687 (+ 5171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:48861: BlockingRecv error: recv got EOF from 127.3.222.3:48861 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 17:47:25 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:25 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:25.141857 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:25.137407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:56203)
0928 17:47:25.137537 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:25.137547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:25.137604 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:25.137764 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:25.137764 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:25.137935 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:25.137986 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:25.137988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.138693 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.138694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:25.139257 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:25.139258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.139606 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.139607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:25.139670 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:25.140113 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:25.140118 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:25.141798 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
W20260928 17:47:25.141956 13126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:25.137455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:56203 (local address 127.3.222.62:46647)
0928 17:47:25.137614 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:25.137616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:25.137626 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:25.137667 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:25.137668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:25.137680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:25.137743 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:25.138069 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.138570 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.139325 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.139518 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.139632 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:25.141883 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:56203: BlockingRecv error: recv got EOF from 127.3.222.1:56203 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":56,"threads_started":1}
Sep 28 17:47:25 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:25 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:25.262768 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:25.258416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:42999)
0928 17:47:25.258525 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:25.258536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:25.258623 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:25.258814 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:25.258815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:25.258974 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:25.259030 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:25.259031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.259729 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.259730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:25.260263 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:25.260265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.260584 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.260585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:25.260636 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:25.261031 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:25.261035 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:25.262657 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
W20260928 17:47:25.262801 13126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:25.258501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:42999 (local address 127.3.222.62:46647)
0928 17:47:25.258581 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:25.258582 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:25.258606 (+ 24us) server_negotiation.cc:407] Connection header received
0928 17:47:25.258715 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:25.258716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:25.258728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:25.258781 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:25.259090 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.259659 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.260335 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.260520 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.260567 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:25.262751 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:42999: BlockingRecv error: recv got EOF from 127.3.222.2:42999 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 17:47:25 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:25 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:25.397576 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:25.393205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:57155)
0928 17:47:25.393339 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:25.393350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:25.393410 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:25.393545 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:25.393546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:25.393710 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:25.393753 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:25.393755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.394395 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.394396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:25.394928 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:25.394930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.395277 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.395279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:25.395351 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:25.395742 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:25.395747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:25.397489 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
W20260928 17:47:25.397644 13126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:25.393336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:57155 (local address 127.3.222.62:46647)
0928 17:47:25.393391 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:25.393393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:25.393404 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:47:25.393475 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:25.393476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:25.393490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:25.393534 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:25.393822 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.394327 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.395016 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:25.395199 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:25.395276 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:25.397590 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:57155: BlockingRecv error: recv got EOF from 127.3.222.3:57155 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 17:47:26 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:26 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:26.146824 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:26.142307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:33711)
0928 17:47:26.142423 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:26.142433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:26.142484 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:26.142621 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:26.142622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:26.142779 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:26.142823 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:26.142824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.143520 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.143521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:26.144155 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:26.144159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.144502 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.144503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:26.144562 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:26.144960 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:26.144965 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:26.146736 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
W20260928 17:47:26.146893 13130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:26.142372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:33711 (local address 127.3.222.62:46647)
0928 17:47:26.142463 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:26.142465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:26.142473 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:26.142536 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:26.142536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:26.142547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:26.142611 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:26.142873 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.143443 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.144230 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.144429 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.144482 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:26.146831 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:33711: BlockingRecv error: recv got EOF from 127.3.222.1:33711 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":42,"threads_started":1}
Sep 28 17:47:26 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:26 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:26.267333 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:26.263237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:36387)
0928 17:47:26.263350 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:26.263361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:26.263418 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:26.263531 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:26.263532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:26.263703 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:26.263747 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:26.263749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.264382 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.264384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:26.264920 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:26.264922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.265231 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.265232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:26.265272 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:26.265667 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:26.265670 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:26.267261 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
W20260928 17:47:26.267406 13130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:26.263280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:36387 (local address 127.3.222.62:46647)
0928 17:47:26.263323 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:26.263324 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:26.263404 (+ 80us) server_negotiation.cc:407] Connection header received
0928 17:47:26.263459 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:26.263459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:26.263469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:26.263520 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:26.263796 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.264300 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.264991 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.265178 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.265226 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:26.267344 (+ 2118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:36387: BlockingRecv error: recv got EOF from 127.3.222.2:36387 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 17:47:26 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:26 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:26.402280 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:26.398023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:34289)
0928 17:47:26.398144 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:26.398153 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:26.398206 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:26.398323 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:26.398324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:26.398483 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:26.398528 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:26.398530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.399143 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.399143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:26.399703 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:26.399705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.400107 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.400108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:26.400171 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:26.400585 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:26.400590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:26.402203 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
W20260928 17:47:26.402361 13130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:26.398075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:34289 (local address 127.3.222.62:46647)
0928 17:47:26.398117 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:26.398120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:26.398202 (+ 82us) server_negotiation.cc:407] Connection header received
0928 17:47:26.398247 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:26.398248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:26.398260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:26.398309 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:26.398578 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.399084 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.399764 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:26.400019 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:26.400083 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:26.402301 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:34289: BlockingRecv error: recv got EOF from 127.3.222.3:34289 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 17:47:27 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:27 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:27.151253 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:27.147272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:57861)
0928 17:47:27.147405 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:27.147415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:27.147466 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:27.147577 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:27.147577 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:27.147729 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:27.147775 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:27.147777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.148383 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.148384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:27.148921 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:27.148923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.149226 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.149226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:27.149272 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:27.149654 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:27.149658 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:27.151178 (+ 1520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
W20260928 17:47:27.151372 13134 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:27.147334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:57861 (local address 127.3.222.62:46647)
0928 17:47:27.147453 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:27.147455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:27.147465 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:27.147506 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:27.147507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:27.147517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:27.147567 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:27.147822 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.148327 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.148977 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.149163 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.149204 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:27.151303 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:57861: BlockingRecv error: recv got EOF from 127.3.222.1:57861 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
Sep 28 17:47:27 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:27 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:27.271940 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:27.267845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:54491)
0928 17:47:27.267968 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:27.267979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:27.268034 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:27.268140 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:27.268141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:27.268314 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:27.268359 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:27.268361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.268969 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.268970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:27.269513 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:27.269515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.269819 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.269820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:27.269854 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:27.270227 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:27.270230 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:27.271854 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
W20260928 17:47:27.271992 13134 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:27.267898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:54491 (local address 127.3.222.62:46647)
0928 17:47:27.267952 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:27.267954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:27.268021 (+ 67us) server_negotiation.cc:407] Connection header received
0928 17:47:27.268074 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:27.268075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:27.268084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:27.268128 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:27.268414 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.268914 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.269582 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.269767 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.269815 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:27.271941 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:54491: BlockingRecv error: recv got EOF from 127.3.222.2:54491 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 17:47:27 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:27 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:27.406769 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:27.402669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:59975)
0928 17:47:27.402811 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:27.402821 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:27.402877 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:27.402983 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:27.402983 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:27.403177 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:27.403250 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:27.403252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.403858 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.403858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:27.404378 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:27.404379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.404656 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.404657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:27.404697 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:27.405119 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:27.405124 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:27.406702 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
W20260928 17:47:27.406839 13134 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:27.402730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:59975 (local address 127.3.222.62:46647)
0928 17:47:27.402798 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:27.402800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:27.402871 (+ 71us) server_negotiation.cc:407] Connection header received
0928 17:47:27.402922 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:27.402923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:27.402934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:27.402977 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:27.403308 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.403794 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.404433 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:27.404606 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:27.404650 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:27.406776 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:59975: BlockingRecv error: recv got EOF from 127.3.222.3:59975 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 17:47:28 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:28 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:28.156064 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:28.151672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:38565)
0928 17:47:28.151810 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:28.151824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:28.151885 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:28.152043 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:28.152045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:28.152211 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:28.152258 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:28.152260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.152905 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.152906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:28.153492 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:28.153495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.153877 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.153877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:28.153927 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:28.154379 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:28.154383 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:28.155995 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260928 17:47:28.156273 13138 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:28.151764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:38565 (local address 127.3.222.62:46647)
0928 17:47:28.151879 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:28.151881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:28.151890 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:28.151929 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:28.151929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:28.151939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:28.151997 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:28.152320 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.152812 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.153582 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.153775 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.153873 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:28.156207 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:38565: BlockingRecv error: recv got EOF from 127.3.222.1:38565 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":46,"threads_started":1}
Sep 28 17:47:28 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:28 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:28.276789 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:28.272375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:40939)
0928 17:47:28.272529 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:28.272542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:28.272602 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:28.272739 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:28.272740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:28.272902 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:28.272949 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:28.272951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.273624 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.273625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:28.274161 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:28.274163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.274504 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.274505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:28.274541 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:28.274923 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:28.274927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:28.276720 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
W20260928 17:47:28.276916 13138 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:28.272503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:40939 (local address 127.3.222.62:46647)
0928 17:47:28.272581 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:28.272583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:28.272591 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:28.272646 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:28.272647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:28.272662 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:28.272711 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:28.273046 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.273549 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.274244 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.274440 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.274497 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:28.276853 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:40939: BlockingRecv error: recv got EOF from 127.3.222.2:40939 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 17:47:28 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:28 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:28.411721 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:28.407190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:36373)
0928 17:47:28.407365 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:28.407377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:28.407441 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:28.407594 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:28.407595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:28.407761 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:28.407804 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:28.407805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.408646 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.408648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:28.409228 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:28.409230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.409555 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.409555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:28.409608 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:28.410020 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:28.410025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:28.411636 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
W20260928 17:47:28.411772 13138 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:28.407305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:36373 (local address 127.3.222.62:46647)
0928 17:47:28.407371 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:28.407373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:28.407434 (+ 61us) server_negotiation.cc:407] Connection header received
0928 17:47:28.407518 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:28.407519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:28.407532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:28.407574 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:28.407865 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.408460 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.409302 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:28.409497 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:28.409552 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:28.411719 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:36373: BlockingRecv error: recv got EOF from 127.3.222.3:36373 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 17:47:29 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:29 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:29.161048 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:29.156531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:38803)
0928 17:47:29.156654 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:29.156664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:29.156721 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:29.156878 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:29.156879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:29.157038 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:29.157084 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:29.157086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.157768 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.157769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:29.158304 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:29.158305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.158654 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.158655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:29.158709 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:29.159265 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:29.159270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:29.160988 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
W20260928 17:47:29.161185 13142 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:29.156659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:38803 (local address 127.3.222.62:46647)
0928 17:47:29.156765 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:29.156767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:29.156775 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:47:29.156797 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:29.156797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:29.156808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:29.156865 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:29.157182 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.157708 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.158370 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.158565 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.158616 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:29.161086 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:38803: BlockingRecv error: recv got EOF from 127.3.222.1:38803 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":45,"threads_started":1}
Sep 28 17:47:29 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:29 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:29.281875 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:29.277317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:46479)
0928 17:47:29.277496 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:29.277509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:29.277587 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:29.277743 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:29.277744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:29.277927 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:29.277974 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:29.277976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.278604 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.278604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:29.279124 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:29.279126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.279528 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.279528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:29.279572 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:29.279973 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:29.279977 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:29.281796 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
W20260928 17:47:29.281926 13142 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:29.277379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:46479 (local address 127.3.222.62:46647)
0928 17:47:29.277429 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:29.277431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:29.277577 (+ 146us) server_negotiation.cc:407] Connection header received
0928 17:47:29.277660 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:29.277661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:29.277673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:29.277729 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:29.278036 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.278539 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.279230 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.279458 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.279517 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:29.281875 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:46479: BlockingRecv error: recv got EOF from 127.3.222.2:46479 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 17:47:29 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:29 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.3@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:29.427495 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:29.412256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.3:57955)
0928 17:47:29.412396 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:29.412406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:29.412464 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:29.412580 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:29.412580 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:29.412742 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:29.412787 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:29.412788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.413412 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.413413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:29.413932 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:29.413934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.414241 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.414242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:29.414276 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:29.425646 (+ 11370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:29.425653 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:29.427434 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
W20260928 17:47:29.427597 13142 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:29.412336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:57955 (local address 127.3.222.62:46647)
0928 17:47:29.412392 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:29.412394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:29.412457 (+ 63us) server_negotiation.cc:407] Connection header received
0928 17:47:29.412504 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:29.412505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:29.412517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:29.412568 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:29.412835 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.413332 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.413996 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:29.414189 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:29.414238 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:29.427526 (+ 13288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.3:57955: BlockingRecv error: recv got EOF from 127.3.222.3:57955 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 17:47:30 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:30 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.1@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:30.166262 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:30.161548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.1:59581)
0928 17:47:30.161717 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:30.161730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:30.161794 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:30.161949 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:30.161950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:30.162110 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:30.162157 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.162159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.162822 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.162823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.163436 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.163438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.163784 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.163784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.163820 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.164337 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:30.164342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:30.166189 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
W20260928 17:47:30.166363 13146 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:30.161679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:59581 (local address 127.3.222.62:46647)
0928 17:47:30.161798 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:30.161800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:30.161811 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:47:30.161838 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:30.161838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:30.161849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:30.161903 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:30.162225 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.162747 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.163517 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.163721 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.163786 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.166271 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.1:59581: BlockingRecv error: recv got EOF from 127.3.222.1:59581 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
Sep 28 17:47:30 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:47:30 dist-test-slave-w3bm krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.222.2@KRBTEST.COM for oryx/127.3.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:47:30.286585 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:30.282386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46647 (local address 127.3.222.2:34111)
0928 17:47:30.282493 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:30.282504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:30.282566 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:30.282691 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:30.282693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:30.282871 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:30.282916 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.282918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.283595 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.283596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.284149 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.284152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.284472 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.284473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.284526 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.284945 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:30.284949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:30.286497 (+ 1548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.222.62:46647: Server oryx/127.3.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
W20260928 17:47:30.286641 13146 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:30.282427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:34111 (local address 127.3.222.62:46647)
0928 17:47:30.282490 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:30.282492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:30.282559 (+ 67us) server_negotiation.cc:407] Connection header received
0928 17:47:30.282618 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:30.282620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:30.282631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:30.282684 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:30.282985 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.283526 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.284223 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.284414 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.284467 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.286590 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.222.2:34111: BlockingRecv error: recv got EOF from 127.3.222.2:34111 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:47:30.335332 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 12670
I20260928 17:47:30.339277 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 12824
I20260928 17:47:30.342973 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 12956
I20260928 17:47:30.346396 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 12600
2026-09-28T17:47:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17301 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T17:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:30Z Disabled control of system clock
I20260928 17:47:30.371829 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38661
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:38661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 17:47:30.450594 13151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:30.450788 13151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:30.450805 13151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:30.451943 13151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:30.451982 13151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:30.451993 13151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:30.452005 13151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:30.452015 13151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:30.453305 13151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:38661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:30.453490 13151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:30.453728 13151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:47:30.459285 13151 server_base.cc:1034] running on GCE node
W20260928 17:47:30.459336 13159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:30.459343 13156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:30.459353 13157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:30.459769 13151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:30.460007 13151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:30.461141 13151 hybrid_clock.cc:648] HybridClock initialized: now 1790617650461130 us; error 28 us; skew 500 ppm
I20260928 17:47:30.462237 13151 webserver.cc:460] Webserver started at http://127.3.222.62:45447/ using document root <none> and password file <none>
I20260928 17:47:30.462455 13151 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:30.462498 13151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:30.462620 13151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:30.463696 13151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "8c4c2e16b93747a3ba80255b8dbbbc79"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "776398655e5f31c80a5fc499503503da"
server_key_iv: "8f3b9200fb1d97d8e3cad4ccfcfd2fd3"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.464012 13151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "8c4c2e16b93747a3ba80255b8dbbbc79"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "776398655e5f31c80a5fc499503503da"
server_key_iv: "8f3b9200fb1d97d8e3cad4ccfcfd2fd3"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.465206 13151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:30.465893 13165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.466037 13151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:30.466109 13151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "8c4c2e16b93747a3ba80255b8dbbbc79"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "776398655e5f31c80a5fc499503503da"
server_key_iv: "8f3b9200fb1d97d8e3cad4ccfcfd2fd3"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.466205 13151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:30.480777 13151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:30.481031 13151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:30.481158 13151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:30.484838 13151 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:38661
I20260928 17:47:30.484866 13217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:38661 every 8 connection(s)
I20260928 17:47:30.485169 13151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:30.485690 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 13151
I20260928 17:47:30.485764 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:30.485764 13218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:30.485947 3960 external_mini_cluster.cc:1428] Setting key 5d49b24f74751be22075eeb37a1f29f0
I20260928 17:47:30.488967 13218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79: Bootstrap starting.
I20260928 17:47:30.489555 13218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:30.489820 13218 log.cc:824] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:30.490470 13218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79: No bootstrap required, opened a new log
I20260928 17:47:30.490968 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:30.486450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.3.222.62:38661)
0928 17:47:30.486611 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:30.486618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:30.486643 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:47:30.487044 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:30.487051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:30.487108 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:30.487281 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:30.487469 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.488302 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.489308 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.489715 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.489797 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.490385 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:30.490389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:30.490389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:30.490550 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:30.490590 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:30.490601 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:30.490648 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:30.490723 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 17:47:30.490780 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
I20260928 17:47:30.492338 13218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4c2e16b93747a3ba80255b8dbbbc79" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38661 } }
I20260928 17:47:30.492463 13218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:30.492484 13218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4c2e16b93747a3ba80255b8dbbbc79, State: Initialized, Role: FOLLOWER
I20260928 17:47:30.492561 13218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4c2e16b93747a3ba80255b8dbbbc79" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38661 } }
I20260928 17:47:30.492623 13218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:30.492642 13218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:30.492673 13218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:30.493137 13218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4c2e16b93747a3ba80255b8dbbbc79" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38661 } }
I20260928 17:47:30.493223 13218 leader_election.cc:302] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c4c2e16b93747a3ba80255b8dbbbc79; no voters:
I20260928 17:47:30.493357 13218 leader_election.cc:288] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:30.493403 13223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:30.493573 13218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:30.493577 13223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [term 1 LEADER]: Becoming Leader. State: Replica: 8c4c2e16b93747a3ba80255b8dbbbc79, State: Running, Role: LEADER
I20260928 17:47:30.493677 13223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4c2e16b93747a3ba80255b8dbbbc79" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38661 } }
I20260928 17:47:30.494024 13225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c4c2e16b93747a3ba80255b8dbbbc79. Latest consensus state: current_term: 1 leader_uuid: "8c4c2e16b93747a3ba80255b8dbbbc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4c2e16b93747a3ba80255b8dbbbc79" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38661 } } }
I20260928 17:47:30.494099 13225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:30.494240 13224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c4c2e16b93747a3ba80255b8dbbbc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4c2e16b93747a3ba80255b8dbbbc79" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38661 } } }
I20260928 17:47:30.494270 13232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:30.494300 13224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:30.494699 13232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:30.496057 13232 catalog_manager.cc:1347] Generated new cluster ID: fc5c9cac637347708d45474bc5650c61
I20260928 17:47:30.496111 13232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:47:30.523379 13232 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:30.523890 13232 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:30.528250 13232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c4c2e16b93747a3ba80255b8dbbbc79: Generated new TSK 0
I20260928 17:47:30.528404 13232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:47:30.537652 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38661
--builtin_ntp_servers=127.3.222.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 17:47:30.614563 13242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:30.614748 13242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:30.614765 13242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:30.615896 13242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:30.615933 13242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:30.615954 13242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:47:30.617252 13242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:30.617461 13242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:30.617728 13242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:30.622267 13250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:30.622293 13247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:30.622310 13248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:30.622447 13242 server_base.cc:1034] running on GCE node
I20260928 17:47:30.622673 13242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:30.622895 13242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:30.624045 13242 hybrid_clock.cc:648] HybridClock initialized: now 1790617650624024 us; error 30 us; skew 500 ppm
I20260928 17:47:30.625092 13242 webserver.cc:460] Webserver started at http://127.3.222.1:36111/ using document root <none> and password file <none>
I20260928 17:47:30.625296 13242 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:30.625341 13242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:30.625463 13242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:30.626457 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "5df9729bee364205870b6978062e307d"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "8a8fac7665c35842a18252877fb365d7"
server_key_iv: "044da854318b421ca56ad29bc0119401"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.626761 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "5df9729bee364205870b6978062e307d"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "8a8fac7665c35842a18252877fb365d7"
server_key_iv: "044da854318b421ca56ad29bc0119401"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.627929 13242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:47:30.628527 13256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.628669 13242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:30.628741 13242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "5df9729bee364205870b6978062e307d"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "8a8fac7665c35842a18252877fb365d7"
server_key_iv: "044da854318b421ca56ad29bc0119401"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.628804 13242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:30.644351 13242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:30.644580 13242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:30.644691 13242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:30.644865 13242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:30.645136 13242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:30.645174 13242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.645198 13242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:30.645216 13242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.650350 13242 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:34973
I20260928 17:47:30.650404 13369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:34973 every 8 connection(s)
I20260928 17:47:30.650662 13242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:47:30.651808 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 13242
I20260928 17:47:30.651876 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:47:30.652033 3960 external_mini_cluster.cc:1428] Setting key a0a5865c4fe972688ba878ad55994ffd
I20260928 17:47:30.652973 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38661
--builtin_ntp_servers=127.3.222.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 17:47:30.655900 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:30.651271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:50495 (local address 127.3.222.62:38661)
0928 17:47:30.651329 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:30.651331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:30.651542 (+ 211us) server_negotiation.cc:407] Connection header received
0928 17:47:30.652068 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:30.652070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:30.652085 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:30.652132 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:30.652821 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.653375 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.654570 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.654768 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.654837 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.655640 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:30.655642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:30.655642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:30.655646 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:30.655662 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:30.655667 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:30.655724 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:30.655783 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 17:47:30.655820 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:47:30.656005 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:30.651273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38661 (local address 127.3.222.1:50495)
0928 17:47:30.651443 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:30.651471 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:30.651975 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:30.652208 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:30.652212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:30.652223 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:30.652744 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.652750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.653455 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.653456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.654485 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.654488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.654852 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.654853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.654981 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.655505 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:30.655506 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:30.655534 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:30.655563 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:30.655725 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:30.655727 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:30.655729 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:47:30.655771 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 17:47:30.655842 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260928 17:47:30.656490 13370 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38661
I20260928 17:47:30.656586 13370 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:30.656775 13370 heartbeater.cc:511] Master 127.3.222.62:38661 requested a full tablet report, sending...
I20260928 17:47:30.657232 13182 ts_manager.cc:194] Registered new tserver with Master: 5df9729bee364205870b6978062e307d (127.3.222.1:34973)
I20260928 17:47:30.657814 13182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.1:50495
W20260928 17:47:30.730468 13373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:30.730657 13373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:30.730676 13373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:30.731830 13373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:30.731868 13373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:30.731900 13373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:47:30.733184 13373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:30.733384 13373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:30.733623 13373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:30.738231 13381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:30.738231 13378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:30.738310 13373 server_base.cc:1034] running on GCE node
W20260928 17:47:30.738297 13379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:30.738622 13373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:30.738829 13373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:30.739955 13373 hybrid_clock.cc:648] HybridClock initialized: now 1790617650739940 us; error 31 us; skew 500 ppm
I20260928 17:47:30.741048 13373 webserver.cc:460] Webserver started at http://127.3.222.2:43313/ using document root <none> and password file <none>
I20260928 17:47:30.741247 13373 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:30.741290 13373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:30.741410 13373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:30.742373 13373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "92280ac09c4141579718ed29143a21fb"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "063f3ee2c9aac44710ea0e41f89c96aa"
server_key_iv: "bb41047189768e524db5730fc3ddf189"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.742703 13373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "92280ac09c4141579718ed29143a21fb"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "063f3ee2c9aac44710ea0e41f89c96aa"
server_key_iv: "bb41047189768e524db5730fc3ddf189"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.743877 13373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:47:30.744458 13387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.744601 13373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:30.744675 13373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "92280ac09c4141579718ed29143a21fb"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "063f3ee2c9aac44710ea0e41f89c96aa"
server_key_iv: "bb41047189768e524db5730fc3ddf189"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.744737 13373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:30.753211 13373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:30.753441 13373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:30.753551 13373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:30.753738 13373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:30.754002 13373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:30.754033 13373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.754058 13373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:30.754076 13373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.759145 13373 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:35219
I20260928 17:47:30.759182 13500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:35219 every 8 connection(s)
I20260928 17:47:30.759482 13373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:47:30.764199 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:30.760237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:53967 (local address 127.3.222.62:38661)
0928 17:47:30.760308 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:30.760311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:30.760416 (+ 105us) server_negotiation.cc:407] Connection header received
0928 17:47:30.760863 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:30.760864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:30.760874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:30.760926 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:30.761512 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.762002 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.762976 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.763161 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.763244 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.763968 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:30.763970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:30.763970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:30.763974 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:30.763989 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:30.763993 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:30.764036 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:30.764112 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 17:47:30.764135 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 17:47:30.764345 13503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:30.760117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38661 (local address 127.3.222.2:53967)
0928 17:47:30.760290 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:30.760329 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:30.760781 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:30.760981 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:30.760985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:30.760992 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:30.761431 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.761435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.762132 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.762133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.762903 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.762905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.763245 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.763246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.763362 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.763849 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:30.763849 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:30.763875 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:30.763896 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:30.764052 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:30.764055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:30.764057 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:47:30.764101 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:47:30.764174 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260928 17:47:30.764770 13501 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38661
I20260928 17:47:30.764849 13501 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:30.765017 13501 heartbeater.cc:511] Master 127.3.222.62:38661 requested a full tablet report, sending...
I20260928 17:47:30.765375 13182 ts_manager.cc:194] Registered new tserver with Master: 92280ac09c4141579718ed29143a21fb (127.3.222.2:35219)
I20260928 17:47:30.765717 13182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.2:53967
I20260928 17:47:30.767289 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 13373
I20260928 17:47:30.767356 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:47:30.767506 3960 external_mini_cluster.cc:1428] Setting key 2c1514c8e380ee6d3ac0246bd2b6bc80
I20260928 17:47:30.768239 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38661
--builtin_ntp_servers=127.3.222.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 17:47:30.844558 13504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:30.844743 13504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:30.844769 13504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:30.845863 13504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:30.845898 13504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:30.845929 13504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:30.847265 13504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:30.847481 13504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:30.847731 13504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:30.852264 13509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:30.852272 13512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:30.852272 13510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:30.852473 13504 server_base.cc:1034] running on GCE node
I20260928 17:47:30.852682 13504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:30.852906 13504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:30.854040 13504 hybrid_clock.cc:648] HybridClock initialized: now 1790617650854030 us; error 30 us; skew 500 ppm
I20260928 17:47:30.855185 13504 webserver.cc:460] Webserver started at http://127.3.222.3:41523/ using document root <none> and password file <none>
I20260928 17:47:30.855422 13504 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:30.855469 13504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:30.855602 13504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:30.856575 13504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "61826b4f000b4e99aab8ccafdc28a23a"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "cd4047b0661fd8813926d0f5f3c0717b"
server_key_iv: "759552c5bc2c747ad851f0473332cdf4"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.856889 13504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "61826b4f000b4e99aab8ccafdc28a23a"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "cd4047b0661fd8813926d0f5f3c0717b"
server_key_iv: "759552c5bc2c747ad851f0473332cdf4"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.857995 13504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:47:30.858592 13518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.858742 13504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:30.858815 13504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "61826b4f000b4e99aab8ccafdc28a23a"
format_stamp: "Formatted at 2026-09-28 17:47:30 on dist-test-slave-w3bm"
server_key: "cd4047b0661fd8813926d0f5f3c0717b"
server_key_iv: "759552c5bc2c747ad851f0473332cdf4"
server_key_version: "encryptionkey@0"
I20260928 17:47:30.858878 13504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:30.877483 13504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:30.877737 13504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:30.877854 13504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:30.878036 13504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:30.878316 13504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:30.878350 13504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.878376 13504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:30.878399 13504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:30.883446 13504 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:43037
I20260928 17:47:30.883505 13631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:43037 every 8 connection(s)
I20260928 17:47:30.883769 13504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:47:30.888532 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:30.884409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:48365 (local address 127.3.222.62:38661)
0928 17:47:30.884466 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:30.884468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:30.884693 (+ 225us) server_negotiation.cc:407] Connection header received
0928 17:47:30.885179 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:30.885180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:30.885191 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:30.885248 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:30.885869 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.886352 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.887321 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.887519 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.887573 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.888277 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:30.888279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:30.888280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:30.888285 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:30.888299 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:30.888303 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:30.888349 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:30.888422 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 17:47:30.888446 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:47:30.888717 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:30.884400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38661 (local address 127.3.222.3:48365)
0928 17:47:30.884580 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:30.884612 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:30.885092 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:30.885316 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:30.885320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:30.885331 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:30.885790 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.885794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.886426 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.886427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.887252 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:30.887255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:30.887589 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:30.887590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:30.887717 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:30.888157 (+ 440us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:30.888157 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:30.888183 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:30.888205 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:30.888354 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:30.888358 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:30.888360 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:47:30.888421 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 17:47:30.888518 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260928 17:47:30.889149 13632 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38661
I20260928 17:47:30.889228 13632 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:30.889366 13632 heartbeater.cc:511] Master 127.3.222.62:38661 requested a full tablet report, sending...
I20260928 17:47:30.889663 13182 ts_manager.cc:194] Registered new tserver with Master: 61826b4f000b4e99aab8ccafdc28a23a (127.3.222.3:43037)
I20260928 17:47:30.889994 13182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.3:48365
I20260928 17:47:30.892313 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 13504
I20260928 17:47:30.892377 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:47:30.892541 3960 external_mini_cluster.cc:1428] Setting key e76a6d9a4c35f2ab130cfadfd9ea5b51
I20260928 17:47:30.893236 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 17:47:30.894868 13221 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:30.894474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.3.222.62:38661)
0928 17:47:30.894512 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:30.894514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:30.894632 (+ 118us) server_negotiation.cc:407] Connection header received
0928 17:47:30.894667 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:30.894668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:30.894676 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:30.894717 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:30.894816 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54666: BlockingRecv error: recv got EOF from 127.0.0.1:54666 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:47:30.894915 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 13242
I20260928 17:47:30.899257 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 13373
I20260928 17:47:30.902798 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 13504
I20260928 17:47:30.906133 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 13151
2026-09-28T17:47:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (558 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T17:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:30Z Disabled control of system clock
I20260928 17:47:30.929556 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36795
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:36795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 17:47:31.005033 13643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:31.005228 13643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:31.005249 13643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:31.006366 13643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:31.006412 13643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:31.006423 13643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:31.006435 13643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:31.006457 13643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:31.007771 13643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:36795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:31.007980 13643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:31.008231 13643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:31.012598 13649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:31.012626 13651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:31.012769 13643 server_base.cc:1034] running on GCE node
W20260928 17:47:31.012626 13648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:31.013031 13643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:31.013280 13643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:31.014420 13643 hybrid_clock.cc:648] HybridClock initialized: now 1790617651014407 us; error 23 us; skew 500 ppm
I20260928 17:47:31.015547 13643 webserver.cc:460] Webserver started at http://127.3.222.62:46171/ using document root <none> and password file <none>
I20260928 17:47:31.015746 13643 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:31.015787 13643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:31.015913 13643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:31.016872 13643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "f2f4ce67c7164b67b0b8800cd1305139"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "1ec93262c6cd9773217e04bd3edd6940"
server_key_iv: "64dc23e41312b30d9fd3bf6cabf71f17"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.017174 13643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "f2f4ce67c7164b67b0b8800cd1305139"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "1ec93262c6cd9773217e04bd3edd6940"
server_key_iv: "64dc23e41312b30d9fd3bf6cabf71f17"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.018265 13643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:47:31.018860 13657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.019012 13643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:31.019093 13643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "f2f4ce67c7164b67b0b8800cd1305139"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "1ec93262c6cd9773217e04bd3edd6940"
server_key_iv: "64dc23e41312b30d9fd3bf6cabf71f17"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.019145 13643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:31.022265 13643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:31.022472 13643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:31.022559 13643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:31.025969 13643 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:36795
I20260928 17:47:31.026010 13709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:36795 every 8 connection(s)
I20260928 17:47:31.026273 13643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:31.026839 13710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:31.029337 13710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139: Bootstrap starting.
I20260928 17:47:31.029834 13710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:31.030086 13710 log.cc:824] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:31.030674 13710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139: No bootstrap required, opened a new log
I20260928 17:47:31.032181 13710 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f4ce67c7164b67b0b8800cd1305139" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36795 } }
I20260928 17:47:31.032305 13710 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:31.032325 13710 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f4ce67c7164b67b0b8800cd1305139, State: Initialized, Role: FOLLOWER
I20260928 17:47:31.032472 13710 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f4ce67c7164b67b0b8800cd1305139" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36795 } }
I20260928 17:47:31.032531 13710 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:31.032550 13710 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:31.032593 13710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:31.032936 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 13643
I20260928 17:47:31.033008 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:31.033109 13710 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f4ce67c7164b67b0b8800cd1305139" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36795 } }
I20260928 17:47:31.033164 3960 external_mini_cluster.cc:1428] Setting key 34e31848ece7bd590b542e9714f7436a
I20260928 17:47:31.033191 13710 leader_election.cc:302] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2f4ce67c7164b67b0b8800cd1305139; no voters:
I20260928 17:47:31.033329 13710 leader_election.cc:288] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:31.033354 13713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:31.033607 13713 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [term 1 LEADER]: Becoming Leader. State: Replica: f2f4ce67c7164b67b0b8800cd1305139, State: Running, Role: LEADER
I20260928 17:47:31.033672 13710 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:31.033703 13713 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f4ce67c7164b67b0b8800cd1305139" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36795 } }
I20260928 17:47:31.034237 13716 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2f4ce67c7164b67b0b8800cd1305139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f4ce67c7164b67b0b8800cd1305139" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36795 } } }
I20260928 17:47:31.034507 13716 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:31.034435 13714 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2f4ce67c7164b67b0b8800cd1305139. Latest consensus state: current_term: 1 leader_uuid: "f2f4ce67c7164b67b0b8800cd1305139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f4ce67c7164b67b0b8800cd1305139" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36795 } } }
I20260928 17:47:31.034765 13714 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:31.034906 13727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:31.035462 13727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:31.035923 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:31.033550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.3.222.62:36795)
0928 17:47:31.033729 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:31.033738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:31.034089 (+ 351us) server_negotiation.cc:407] Connection header received
0928 17:47:31.034314 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:31.034322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:31.034372 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:31.034478 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:31.035492 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:31.035495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:31.035496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:31.035500 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:31.035545 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:31.035559 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:31.035624 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:31.035687 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 17:47:31.035755 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260928 17:47:31.037012 13727 catalog_manager.cc:1347] Generated new cluster ID: 5849588a52ae4c9399e2c6fc6be24783
I20260928 17:47:31.037051 13727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:47:31.045431 13727 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:31.045564 13727 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:31.054175 13727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2f4ce67c7164b67b0b8800cd1305139: Generated new TSK 0
I20260928 17:47:31.054349 13727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:47:31.063844 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36795
--builtin_ntp_servers=127.3.222.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 17:47:31.159749 13734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:31.159945 13734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:31.159974 13734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:31.161064 13734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:31.161099 13734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:31.161132 13734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:47:31.162425 13734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:31.162642 13734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:31.162912 13734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:31.167721 13739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:31.167851 13742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:31.167907 13734 server_base.cc:1034] running on GCE node
W20260928 17:47:31.167757 13740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:31.168191 13734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:31.168395 13734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:31.169545 13734 hybrid_clock.cc:648] HybridClock initialized: now 1790617651169522 us; error 37 us; skew 500 ppm
I20260928 17:47:31.170660 13734 webserver.cc:460] Webserver started at http://127.3.222.1:40907/ using document root <none> and password file <none>
I20260928 17:47:31.170886 13734 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:31.170928 13734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:31.171039 13734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:31.172137 13734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "3bc826d419ad4411a811552651f1551a"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "130b5b6c9453d85ca246261d57704cf7"
server_key_iv: "3142e414caae2197e994a4086739f891"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.172446 13734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "3bc826d419ad4411a811552651f1551a"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "130b5b6c9453d85ca246261d57704cf7"
server_key_iv: "3142e414caae2197e994a4086739f891"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.173607 13734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:31.174284 13748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.174484 13734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:31.174566 13734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "3bc826d419ad4411a811552651f1551a"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "130b5b6c9453d85ca246261d57704cf7"
server_key_iv: "3142e414caae2197e994a4086739f891"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.174619 13734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:31.177774 13734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:31.177975 13734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:31.178061 13734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:31.178210 13734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:31.178476 13734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:31.178508 13734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.178526 13734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:31.178542 13734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.183579 13734 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:46051
I20260928 17:47:31.183637 13861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:46051 every 8 connection(s)
I20260928 17:47:31.183902 13734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:47:31.186544 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:31.184484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:35783 (local address 127.3.222.62:36795)
0928 17:47:31.184523 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:31.184525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:31.184770 (+ 245us) server_negotiation.cc:407] Connection header received
0928 17:47:31.185185 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:31.185187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:31.185193 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:31.185257 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:31.186324 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:31.186325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:31.186325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:31.186329 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:31.186344 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:31.186348 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:31.186398 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:31.186477 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 17:47:31.186493 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:47:31.186650 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:31.184463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36795 (local address 127.3.222.1:35783)
0928 17:47:31.184631 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:31.184669 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:31.185079 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:31.185341 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:31.185344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:31.185352 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:31.186188 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:31.186189 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:31.186215 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:31.186241 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:31.186396 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:31.186398 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:31.186400 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:47:31.186450 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 17:47:31.186494 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260928 17:47:31.187072 13862 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36795
I20260928 17:47:31.187147 13862 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:31.187317 13862 heartbeater.cc:511] Master 127.3.222.62:36795 requested a full tablet report, sending...
I20260928 17:47:31.187790 13674 ts_manager.cc:194] Registered new tserver with Master: 3bc826d419ad4411a811552651f1551a (127.3.222.1:46051)
I20260928 17:47:31.188536 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 13734
I20260928 17:47:31.188619 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:47:31.188789 3960 external_mini_cluster.cc:1428] Setting key 39217146be79f276886c0c377d5a66dd
I20260928 17:47:31.189534 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36795
--builtin_ntp_servers=127.3.222.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 17:47:31.266196 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:31.266376 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:31.266395 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:31.267475 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:31.267508 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:31.267531 13865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:47:31.268745 13865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:31.268915 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:31.269122 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:31.273860 13871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:31.273917 13865 server_base.cc:1034] running on GCE node
W20260928 17:47:31.273866 13873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:31.274755 13870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:31.274973 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:31.275172 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:31.276319 13865 hybrid_clock.cc:648] HybridClock initialized: now 1790617651276310 us; error 28 us; skew 500 ppm
I20260928 17:47:31.277475 13865 webserver.cc:460] Webserver started at http://127.3.222.2:42321/ using document root <none> and password file <none>
I20260928 17:47:31.277660 13865 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:31.277705 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:31.277840 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:31.278848 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "9cbb5ea825ec46da876b7a2c122d1752"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "4769d07a127fe53f285408c6a37e24ce"
server_key_iv: "d5507026bf65087f2be80d857581c786"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.279155 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "9cbb5ea825ec46da876b7a2c122d1752"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "4769d07a127fe53f285408c6a37e24ce"
server_key_iv: "d5507026bf65087f2be80d857581c786"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.280305 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:47:31.280910 13879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.281057 13865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:47:31.281131 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "9cbb5ea825ec46da876b7a2c122d1752"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "4769d07a127fe53f285408c6a37e24ce"
server_key_iv: "d5507026bf65087f2be80d857581c786"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.281193 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:31.284281 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:31.284489 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:31.284596 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:31.284777 13865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:31.285029 13865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:31.285058 13865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.285082 13865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:31.285106 13865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.290139 13865 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:38657
I20260928 17:47:31.290189 13992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:38657 every 8 connection(s)
I20260928 17:47:31.290436 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:47:31.292872 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:31.291005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:37037 (local address 127.3.222.62:36795)
0928 17:47:31.291051 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:31.291053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:31.291266 (+ 213us) server_negotiation.cc:407] Connection header received
0928 17:47:31.291699 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:31.291700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:31.291706 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:31.291754 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:31.292690 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:31.292692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:31.292692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:31.292695 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:31.292709 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:31.292713 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:31.292755 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:31.292813 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 17:47:31.292824 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:31.293025 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:31.290994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36795 (local address 127.3.222.2:37037)
0928 17:47:31.291156 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:31.291185 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:31.291597 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:31.291819 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:31.291822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:31.291829 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:31.292572 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:31.292573 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:31.292595 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:31.292619 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:31.292767 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:31.292769 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:31.292771 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:47:31.292814 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:47:31.292856 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260928 17:47:31.293406 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 13865
I20260928 17:47:31.293438 13993 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36795
I20260928 17:47:31.293504 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:47:31.293506 13993 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:31.293632 13993 heartbeater.cc:511] Master 127.3.222.62:36795 requested a full tablet report, sending...
I20260928 17:47:31.293711 3960 external_mini_cluster.cc:1428] Setting key 6d43fa503855cf15027e22ec89540ee4
I20260928 17:47:31.293951 13674 ts_manager.cc:194] Registered new tserver with Master: 9cbb5ea825ec46da876b7a2c122d1752 (127.3.222.2:38657)
I20260928 17:47:31.294597 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36795
--builtin_ntp_servers=127.3.222.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 17:47:31.372035 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:31.372227 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:31.372255 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:31.373349 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:31.373384 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:31.373417 13996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:31.374682 13996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:31.374897 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:31.375146 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:31.379951 14001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:31.380020 13996 server_base.cc:1034] running on GCE node
W20260928 17:47:31.380110 14004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:31.380020 14002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:31.380366 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:31.380549 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:31.381685 13996 hybrid_clock.cc:648] HybridClock initialized: now 1790617651381674 us; error 26 us; skew 500 ppm
I20260928 17:47:31.382815 13996 webserver.cc:460] Webserver started at http://127.3.222.3:33915/ using document root <none> and password file <none>
I20260928 17:47:31.383006 13996 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:31.383049 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:31.383181 13996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:31.384207 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "ff1bb42d72a54040bcafb23d06d86746"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "d152eb7e1208919443f19c20c4af72dc"
server_key_iv: "5a8864c9355109ae4d4e6a48ae65b0cb"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.384532 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "ff1bb42d72a54040bcafb23d06d86746"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "d152eb7e1208919443f19c20c4af72dc"
server_key_iv: "5a8864c9355109ae4d4e6a48ae65b0cb"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.385680 13996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:31.386271 14010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.386497 13996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:31.386581 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "ff1bb42d72a54040bcafb23d06d86746"
format_stamp: "Formatted at 2026-09-28 17:47:31 on dist-test-slave-w3bm"
server_key: "d152eb7e1208919443f19c20c4af72dc"
server_key_iv: "5a8864c9355109ae4d4e6a48ae65b0cb"
server_key_version: "encryptionkey@0"
I20260928 17:47:31.386646 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:31.389915 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:31.390136 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:31.390247 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:31.390420 13996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:31.390684 13996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:31.390714 13996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.390733 13996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:31.390755 13996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:31.395889 13996 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:46031
I20260928 17:47:31.395925 14123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:46031 every 8 connection(s)
I20260928 17:47:31.396202 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:47:31.398510 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 13996
I20260928 17:47:31.398589 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:47:31.398756 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:31.396812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:43773 (local address 127.3.222.62:36795)
0928 17:47:31.396873 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:31.396876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:31.397059 (+ 183us) server_negotiation.cc:407] Connection header received
0928 17:47:31.397490 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:31.397491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:31.397496 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:31.397542 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:31.398523 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:31.398524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:31.398525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:47:31.398529 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:31.398544 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:47:31.398548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:31.398596 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:31.398666 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 17:47:31.398680 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:47:31.398782 3960 external_mini_cluster.cc:1428] Setting key fb78c1543822bbbe69dbb60aee8558f6
I20260928 17:47:31.398957 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:31.396797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36795 (local address 127.3.222.3:43773)
0928 17:47:31.396965 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:31.396995 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:31.397415 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:31.397593 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:31.397596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:31.397603 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:31.398387 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:47:31.398387 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:31.398412 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:47:31.398438 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:31.398598 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:31.398603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:31.398606 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:31.398671 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 17:47:31.398751 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260928 17:47:31.399483 14124 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36795
I20260928 17:47:31.399569 14124 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:31.399705 14124 heartbeater.cc:511] Master 127.3.222.62:36795 requested a full tablet report, sending...
I20260928 17:47:31.400002 13673 ts_manager.cc:194] Registered new tserver with Master: ff1bb42d72a54040bcafb23d06d86746 (127.3.222.3:46031)
I20260928 17:47:31.410256 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 17:47:31.411947 13717 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:47:31.411509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.3.222.62:36795)
0928 17:47:31.411558 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:31.411560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:31.411663 (+ 103us) server_negotiation.cc:407] Connection header received
0928 17:47:31.411732 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:31.411733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:31.411738 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:31.411782 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:31.411892 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48830: BlockingRecv error: recv got EOF from 127.0.0.1:48830 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:47:31.412024 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 13734
I20260928 17:47:31.416177 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 13865
I20260928 17:47:31.420097 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 13996
I20260928 17:47:31.423895 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 13643
2026-09-28T17:47:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (518 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:47:31 dist-test-slave-w3bm krb5kdc[14134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:47:31 dist-test-slave-w3bm krb5kdc[14134](info): set up 2 sockets
Sep 28 17:47:31 dist-test-slave-w3bm krb5kdc[14134](info): commencing operation
krb5kdc: starting...
W20260928 17:47:32.379714 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:47:32 dist-test-slave-w3bm krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:47:32.492161 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:45067
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:45067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:32.568645 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:32.568836 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:32.568863 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:32.569942 14147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:47:32.569979 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:32.569998 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:32.570013 14147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:47:32.570029 14147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:47:32.571452 14147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:45067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:45067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:32.571671 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:32.571858 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:32.576321 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:32.576329 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:32.576342 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:32.576413 14147 server_base.cc:1034] running on GCE node
I20260928 17:47:32.576711 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:32.576961 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:32.578089 14147 hybrid_clock.cc:648] HybridClock initialized: now 1790617652578077 us; error 26 us; skew 500 ppm
Sep 28 17:47:32 dist-test-slave-w3bm krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:32.580279 14147 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:47:32.580655 14147 webserver.cc:460] Webserver started at http://127.3.222.62:46579/ using document root <none> and password file <none>
I20260928 17:47:32.580835 14147 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:32.580879 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:32.580991 14147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:32.581935 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "a77129fcbbee4902b9b81e11944bc2a7"
format_stamp: "Formatted at 2026-09-28 17:47:32 on dist-test-slave-w3bm"
server_key: "ddf4257401d5345033d3d90d57a152de"
server_key_iv: "3bf4893ca187580b00b7f4042aaaa370"
server_key_version: "encryptionkey@0"
I20260928 17:47:32.582245 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "a77129fcbbee4902b9b81e11944bc2a7"
format_stamp: "Formatted at 2026-09-28 17:47:32 on dist-test-slave-w3bm"
server_key: "ddf4257401d5345033d3d90d57a152de"
server_key_iv: "3bf4893ca187580b00b7f4042aaaa370"
server_key_version: "encryptionkey@0"
I20260928 17:47:32.583393 14147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:47:32.584079 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:32.584251 14147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:47:32.584323 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "a77129fcbbee4902b9b81e11944bc2a7"
format_stamp: "Formatted at 2026-09-28 17:47:32 on dist-test-slave-w3bm"
server_key: "ddf4257401d5345033d3d90d57a152de"
server_key_iv: "3bf4893ca187580b00b7f4042aaaa370"
server_key_version: "encryptionkey@0"
I20260928 17:47:32.584379 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:32.624090 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:32.626152 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:32.626288 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:32.629829 14147 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:45067
I20260928 17:47:32.629874 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:45067 every 8 connection(s)
I20260928 17:47:32.630187 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:47:32.631083 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:32.633258 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7: Bootstrap starting.
I20260928 17:47:32.633782 14215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:32.634027 14215 log.cc:824] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:32.634635 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7: No bootstrap required, opened a new log
I20260928 17:47:32.636062 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 14147
I20260928 17:47:32.636158 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:47:32.636330 3960 external_mini_cluster.cc:1428] Setting key f7de0f5e2bff1e7a19f9f3277d8b78f4
I20260928 17:47:32.636312 14215 raft_consensus.cc:348] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77129fcbbee4902b9b81e11944bc2a7" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 45067 } }
I20260928 17:47:32.636476 14215 raft_consensus.cc:374] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:32.636519 14215 raft_consensus.cc:729] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77129fcbbee4902b9b81e11944bc2a7, State: Initialized, Role: FOLLOWER
I20260928 17:47:32.636648 14215 consensus_queue.cc:260] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77129fcbbee4902b9b81e11944bc2a7" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 45067 } }
I20260928 17:47:32.636727 14215 raft_consensus.cc:388] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:47:32.636785 14215 raft_consensus.cc:482] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:47:32.636839 14215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:32.637378 14215 raft_consensus.cc:504] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77129fcbbee4902b9b81e11944bc2a7" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 45067 } }
I20260928 17:47:32.637458 14215 leader_election.cc:302] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a77129fcbbee4902b9b81e11944bc2a7; no voters:
I20260928 17:47:32.637594 14215 leader_election.cc:288] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:47:32.637629 14220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:32.637763 14220 raft_consensus.cc:686] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [term 1 LEADER]: Becoming Leader. State: Replica: a77129fcbbee4902b9b81e11944bc2a7, State: Running, Role: LEADER
I20260928 17:47:32.637826 14215 sys_catalog.cc:564] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:47:32.637877 14220 consensus_queue.cc:237] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77129fcbbee4902b9b81e11944bc2a7" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 45067 } }
I20260928 17:47:32.638221 14220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77129fcbbee4902b9b81e11944bc2a7. Latest consensus state: current_term: 1 leader_uuid: "a77129fcbbee4902b9b81e11944bc2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77129fcbbee4902b9b81e11944bc2a7" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 45067 } } }
I20260928 17:47:32.638273 14222 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a77129fcbbee4902b9b81e11944bc2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77129fcbbee4902b9b81e11944bc2a7" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 45067 } } }
I20260928 17:47:32.638355 14220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:32.638361 14222 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:47:32.638552 14229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:47:32.639281 14229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:47:32.641124 14229 catalog_manager.cc:1347] Generated new cluster ID: c8e30161a3e245d5a96671fe87deb6c2
I20260928 17:47:32.641170 14229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:47:32 dist-test-slave-w3bm krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:32.647290 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:32.636751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.3.222.62:45067)
0928 17:47:32.636914 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:32.636923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:32.637001 (+ 78us) server_negotiation.cc:407] Connection header received
0928 17:47:32.637352 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:32.637358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:32.637401 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:32.637528 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:32.638302 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.639793 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.640595 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.641174 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.641275 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:32.643949 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:32.643954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:32.643954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:32.643966 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:32.645793 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:32.646136 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:32.646140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:32.646140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:32.646251 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:32.646414 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:32.646415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:32.646415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:32.646724 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:32.646877 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:32.646944 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 17:47:32.647074 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260928 17:47:32.650914 14229 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:47:32.651588 14229 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:47:32.654740 14229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7: Generated new TSK 0
I20260928 17:47:32.654881 14229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:47:32.690361 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:45067
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:32.767992 14241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:32.768188 14241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:32.768208 14241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:32.769277 14241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:32.769312 14241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:32.769336 14241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:47:32.770697 14241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:32.770910 14241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:32.771113 14241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:32.775557 14246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:32.775586 14249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:32.775533 14247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:32.775664 14241 server_base.cc:1034] running on GCE node
I20260928 17:47:32.775959 14241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:32.776178 14241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:32.777302 14241 hybrid_clock.cc:648] HybridClock initialized: now 1790617652777294 us; error 25 us; skew 500 ppm
Sep 28 17:47:32 dist-test-slave-w3bm krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:32.779553 14241 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:47:32.779924 14241 webserver.cc:460] Webserver started at http://127.3.222.1:36177/ using document root <none> and password file <none>
I20260928 17:47:32.780118 14241 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:32.780189 14241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:32.780305 14241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:32.781273 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "957a1c9ee8ef4a1190c6882b05da5675"
format_stamp: "Formatted at 2026-09-28 17:47:32 on dist-test-slave-w3bm"
server_key: "d4ae192cf34c329ab81f77e473b74c9c"
server_key_iv: "fda0023cdc472079cfb24e99a6692dfc"
server_key_version: "encryptionkey@0"
I20260928 17:47:32.781571 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "957a1c9ee8ef4a1190c6882b05da5675"
format_stamp: "Formatted at 2026-09-28 17:47:32 on dist-test-slave-w3bm"
server_key: "d4ae192cf34c329ab81f77e473b74c9c"
server_key_iv: "fda0023cdc472079cfb24e99a6692dfc"
server_key_version: "encryptionkey@0"
I20260928 17:47:32.782701 14241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:47:32.783360 14256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:32.783484 14241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:32.783553 14241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "957a1c9ee8ef4a1190c6882b05da5675"
format_stamp: "Formatted at 2026-09-28 17:47:32 on dist-test-slave-w3bm"
server_key: "d4ae192cf34c329ab81f77e473b74c9c"
server_key_iv: "fda0023cdc472079cfb24e99a6692dfc"
server_key_version: "encryptionkey@0"
I20260928 17:47:32.783618 14241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:32.791697 14241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:32.793442 14241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:32.793547 14241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:32.793721 14241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:32.793963 14241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:32.793995 14241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:32.794013 14241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:32.794023 14241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:32.799153 14241 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:38161
I20260928 17:47:32.799186 14369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:38161 every 8 connection(s)
I20260928 17:47:32.799481 14241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:47:32.804606 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 14241
Sep 28 17:47:32 dist-test-slave-w3bm krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:32.804680 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:47:32.804881 3960 external_mini_cluster.cc:1428] Setting key fe843306d96618b092355dce599d66b6
I20260928 17:47:32.807476 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:32.800060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:33345 (local address 127.3.222.62:45067)
0928 17:47:32.800119 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:32.800121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:32.800271 (+ 150us) server_negotiation.cc:407] Connection header received
0928 17:47:32.800709 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:32.800711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:32.800723 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:32.800777 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:32.801634 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.802127 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.803039 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.803287 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.803343 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:32.805136 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:32.805137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:32.805138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:32.805145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:32.806301 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:32.806639 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:32.806640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:32.806640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:32.806690 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:32.806891 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:32.806892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:32.806893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:32.806998 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:32.807051 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:32.807310 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 17:47:32.807391 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:47:32.807621 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:32.800028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:45067 (local address 127.3.222.1:33345)
0928 17:47:32.800172 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:32.800202 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:32.800626 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:32.800839 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:32.800843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:32.801376 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:32.801548 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:32.801553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.802204 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.802205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:32.802963 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:32.802965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.803350 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.803351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:32.803458 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:32.803796 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:32.803803 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:32.805055 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:32.806368 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:32.806374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:32.806376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:32.806576 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:32.806742 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:32.806743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:32.806743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:32.806840 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:32.807051 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:32.807057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:32.807178 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 17:47:32.807289 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 17:47:32.807435 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260928 17:47:32.807972 14370 heartbeater.cc:347] Connected to a master server at 127.3.222.62:45067
I20260928 17:47:32.808048 14370 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:32.808195 14370 heartbeater.cc:511] Master 127.3.222.62:45067 requested a full tablet report, sending...
I20260928 17:47:32.808617 14179 ts_manager.cc:194] Registered new tserver with Master: 957a1c9ee8ef4a1190c6882b05da5675 (127.3.222.1:38161)
I20260928 17:47:32.809181 14179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:33345
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:47:32.829906 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:45067
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:32.905961 14375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:32.906152 14375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:32.906172 14375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:32.907228 14375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:32.907261 14375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:32.907284 14375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:47:32.908608 14375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:32.908771 14375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:32.908926 14375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:47:32.913473 14380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:32.913609 14383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:32.913583 14381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:32.913666 14375 server_base.cc:1034] running on GCE node
I20260928 17:47:32.913908 14375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:32.914119 14375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:32.915293 14375 hybrid_clock.cc:648] HybridClock initialized: now 1790617652915273 us; error 32 us; skew 500 ppm
Sep 28 17:47:32 dist-test-slave-w3bm krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:32.917558 14375 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:47:32.917935 14375 webserver.cc:460] Webserver started at http://127.3.222.2:37797/ using document root <none> and password file <none>
I20260928 17:47:32.918118 14375 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:32.918162 14375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:32.918278 14375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:32.919231 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "29c903f0a0ad49d493049bdb8fc98c11"
format_stamp: "Formatted at 2026-09-28 17:47:32 on dist-test-slave-w3bm"
server_key: "62c0d568c06fbf1e29dfbafbcd6a122e"
server_key_iv: "ed6979be94c3c84372fcb7290171d098"
server_key_version: "encryptionkey@0"
I20260928 17:47:32.919561 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "29c903f0a0ad49d493049bdb8fc98c11"
format_stamp: "Formatted at 2026-09-28 17:47:32 on dist-test-slave-w3bm"
server_key: "62c0d568c06fbf1e29dfbafbcd6a122e"
server_key_iv: "ed6979be94c3c84372fcb7290171d098"
server_key_version: "encryptionkey@0"
I20260928 17:47:32.920747 14375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:32.921561 14390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:32.921722 14375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:47:32.921806 14375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "29c903f0a0ad49d493049bdb8fc98c11"
format_stamp: "Formatted at 2026-09-28 17:47:32 on dist-test-slave-w3bm"
server_key: "62c0d568c06fbf1e29dfbafbcd6a122e"
server_key_iv: "ed6979be94c3c84372fcb7290171d098"
server_key_version: "encryptionkey@0"
I20260928 17:47:32.921878 14375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:32.933081 14375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:32.934623 14375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:32.934739 14375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:32.934929 14375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:32.935181 14375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:32.935243 14375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:32.935279 14375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:32.935303 14375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:32.940550 14375 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:41117
I20260928 17:47:32.940601 14503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:41117 every 8 connection(s)
I20260928 17:47:32.940881 14375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:47:32.943607 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 14375
I20260928 17:47:32.943691 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:47:32.943866 3960 external_mini_cluster.cc:1428] Setting key 48eaff42ea45953403f590d1e7403804
Sep 28 17:47:32 dist-test-slave-w3bm krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:32.949815 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:32.941433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:40859 (local address 127.3.222.62:45067)
0928 17:47:32.941474 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:32.941476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:32.941667 (+ 191us) server_negotiation.cc:407] Connection header received
0928 17:47:32.942164 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:32.942165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:32.942178 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:32.942227 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:32.943107 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.943774 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.944916 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.945175 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.945235 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:32.947356 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:32.947357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:32.947358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:32.947365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:32.948572 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:32.948998 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:32.948999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:32.948999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:32.949042 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:32.949277 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:32.949278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:32.949278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:32.949367 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:32.949409 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:32.949697 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 17:47:32.949755 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:47:32.950052 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:32.941416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:45067 (local address 127.3.222.2:40859)
0928 17:47:32.941559 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:32.941592 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:32.942074 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:32.942291 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:32.942295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:32.942831 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:32.943015 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:32.943020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.943865 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.943866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:32.944841 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:32.944844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:32.945245 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:32.945245 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:32.945334 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:32.945797 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:32.945807 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:32.947277 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:32.948646 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:32.948652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:32.948654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:32.948932 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:32.949097 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:32.949098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:32.949098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:32.949223 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:32.949418 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:32.949422 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:32.949581 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 17:47:32.949690 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 17:47:32.949841 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260928 17:47:32.950501 14504 heartbeater.cc:347] Connected to a master server at 127.3.222.62:45067
I20260928 17:47:32.950591 14504 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:32.950760 14504 heartbeater.cc:511] Master 127.3.222.62:45067 requested a full tablet report, sending...
I20260928 17:47:32.951097 14179 ts_manager.cc:194] Registered new tserver with Master: 29c903f0a0ad49d493049bdb8fc98c11 (127.3.222.2:41117)
I20260928 17:47:32.951480 14179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:40859
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:47:32.969790 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:45067
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:47:33.049361 14509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:47:33.049558 14509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:47:33.049579 14509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:47:33.050715 14509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:47:33.050756 14509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:47:33.050791 14509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:47:33.052232 14509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:47:33.052443 14509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:47:33.052639 14509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:47:33.057552 14509 server_base.cc:1034] running on GCE node
W20260928 17:47:33.057552 14517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:33.057585 14514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:47:33.058080 14515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:47:33.058301 14509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:47:33.058534 14509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:47:33.059724 14509 hybrid_clock.cc:648] HybridClock initialized: now 1790617653059711 us; error 25 us; skew 500 ppm
Sep 28 17:47:33 dist-test-slave-w3bm krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617653, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:47:33.062216 14509 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:47:33.062630 14509 webserver.cc:460] Webserver started at http://127.3.222.3:42053/ using document root <none> and password file <none>
I20260928 17:47:33.062817 14509 fs_manager.cc:359] Metadata directory not provided
I20260928 17:47:33.062860 14509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:47:33.062989 14509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:47:33.064147 14509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "91dafdc1d2984394b0eb2e27ba164024"
format_stamp: "Formatted at 2026-09-28 17:47:33 on dist-test-slave-w3bm"
server_key: "649fcf08283ecf4519b8f6d56b04317b"
server_key_iv: "2b66c6c3dbae18bc5b3d29ec1d33e886"
server_key_version: "encryptionkey@0"
I20260928 17:47:33.064478 14509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "91dafdc1d2984394b0eb2e27ba164024"
format_stamp: "Formatted at 2026-09-28 17:47:33 on dist-test-slave-w3bm"
server_key: "649fcf08283ecf4519b8f6d56b04317b"
server_key_iv: "2b66c6c3dbae18bc5b3d29ec1d33e886"
server_key_version: "encryptionkey@0"
I20260928 17:47:33.065722 14509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:47:33.066512 14524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:33.066682 14509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:47:33.066802 14509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "91dafdc1d2984394b0eb2e27ba164024"
format_stamp: "Formatted at 2026-09-28 17:47:33 on dist-test-slave-w3bm"
server_key: "649fcf08283ecf4519b8f6d56b04317b"
server_key_iv: "2b66c6c3dbae18bc5b3d29ec1d33e886"
server_key_version: "encryptionkey@0"
I20260928 17:47:33.066865 14509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:47:33.099611 14509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:47:33.101413 14509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:47:33.101548 14509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:47:33.101759 14509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:47:33.102038 14509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:47:33.102072 14509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:33.102098 14509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:47:33.102123 14509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:47:33.107354 14509 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:43537
I20260928 17:47:33.107414 14637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:43537 every 8 connection(s)
I20260928 17:47:33.107686 14509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:47:33 dist-test-slave-w3bm krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617653, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:47:33.114153 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 14509
I20260928 17:47:33.114264 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:47:33.114466 3960 external_mini_cluster.cc:1428] Setting key 4eb5e5220214e56f3392dcff412e1b51
I20260928 17:47:33.116000 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.108340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:37725 (local address 127.3.222.62:45067)
0928 17:47:33.108381 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:33.108383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:33.108577 (+ 194us) server_negotiation.cc:407] Connection header received
0928 17:47:33.108994 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:33.108995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:33.109006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:33.109061 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:33.109911 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.110422 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.111374 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.111553 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.111604 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.113420 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:33.113421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:33.113422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:33.113428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:33.114651 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:33.115112 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:33.115114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:33.115114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:33.115164 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:33.115417 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:33.115418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:33.115419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:33.115522 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:33.115575 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:33.115838 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 17:47:33.115916 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:33.116148 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.108294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:45067 (local address 127.3.222.3:37725)
0928 17:47:33.108432 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:33.108479 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:33.108920 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:33.109114 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:33.109117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:33.109641 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:47:33.109820 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.109824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.110493 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.110494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.111302 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.111305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.111610 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.111611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.111698 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.112132 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:47:33.112139 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:47:33.113343 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:47:33.114720 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:33.114726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:33.114728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:33.115023 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:33.115239 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:47:33.115239 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:47:33.115240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:47:33.115362 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:47:33.115575 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:47:33.115581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:47:33.115707 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 17:47:33.115810 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 17:47:33.115949 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260928 17:47:33.116532 14638 heartbeater.cc:347] Connected to a master server at 127.3.222.62:45067
I20260928 17:47:33.116604 14638 heartbeater.cc:464] Registering TS with master...
I20260928 17:47:33.116731 14638 heartbeater.cc:511] Master 127.3.222.62:45067 requested a full tablet report, sending...
I20260928 17:47:33.117026 14179 ts_manager.cc:194] Registered new tserver with Master: 91dafdc1d2984394b0eb2e27ba164024 (127.3.222.3:43537)
I20260928 17:47:33.117373 14179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:37725
I20260928 17:47:33.125878 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:47:33.132143 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.127096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.3.222.62:45067)
0928 17:47:33.127142 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:33.127143 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:33.127283 (+ 140us) server_negotiation.cc:407] Connection header received
0928 17:47:33.127330 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:33.127331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:33.127339 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:33.127381 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:47:33.127853 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.128299 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.129036 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.129204 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.129246 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.130252 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:47:33.130253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:47:33.130253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:47:33.130258 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:47:33.131327 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:33.131604 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:33.131604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:33.131605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:33.131645 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:47:33.131767 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:47:33.131767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:47:33.131768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:47:33.131885 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:47:33.131949 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:33.132026 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 17:47:33.132079 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:47:33.133456 14179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:47:33.134938 14179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:47:33.140833 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.136948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:43537 (local address 127.0.0.1:57846)
0928 17:47:33.137233 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:33.137241 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:33.137283 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:33.137433 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:33.137434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:33.137436 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:33.137478 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.137480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.138476 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.138477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.139733 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.139736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.140551 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.140552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.140622 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.140638 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 17:47:33.140704 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 17:47:33.140752 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":30,"threads_started":1}
I20260928 17:47:33.140842 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.137040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.3.222.3:43537)
0928 17:47:33.137224 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:33.137228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:33.137276 (+ 48us) server_negotiation.cc:407] Connection header received
0928 17:47:33.137338 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:33.137340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:33.137353 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:33.137418 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:33.137549 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.138381 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.139801 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.140472 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.140540 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.140592 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:33.140723 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 17:47:33.140768 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260928 17:47:33.141037 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.136864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.3.222.2:41117)
0928 17:47:33.137316 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:33.137320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:33.137329 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:47:33.137363 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:33.137365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:33.137378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:33.137445 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:33.137645 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.138795 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.140024 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.140491 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.140553 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.140607 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:33.140871 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 17:47:33.140964 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":72,"threads_started":1}
I20260928 17:47:33.141235 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.136864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.3.222.1:38161)
0928 17:47:33.137066 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:33.137071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:33.137189 (+ 118us) server_negotiation.cc:407] Connection header received
0928 17:47:33.137273 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:33.137276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:33.137292 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:33.137525 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:33.137595 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.138770 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.140067 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.140731 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.141058 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.141107 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:33.141132 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 17:47:33.141169 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260928 17:47:33.141010 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.136767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:38161 (local address 127.0.0.1:48278)
0928 17:47:33.137102 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:33.137144 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:33.137196 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:33.137390 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:33.137392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:33.137395 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:33.137458 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.137460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.138885 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.138886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.139987 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.139989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.140818 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.140819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.140876 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.140881 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:47:33.140909 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 17:47:33.140944 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":37,"spinlock_wait_cycles":384,"thread_start_us":62,"threads_started":1}
I20260928 17:47:33.140834 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.136764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:41117 (local address 127.0.0.1:45350)
0928 17:47:33.137038 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:33.137077 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:33.137142 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:33.137544 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:33.137545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:33.137547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:33.137576 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.137578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.138877 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.138878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.139958 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.139960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.140637 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.140637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.140687 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.140690 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:47:33.140730 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 17:47:33.140762 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":29,"thread_start_us":61,"threads_started":1}
I20260928 17:47:33.142084 14572 tablet_service.cc:1467] Processing CreateTablet for tablet b0da7cee023c4066b112aae12879949f (DEFAULT_TABLE table=test-table [id=ee725fa1cef340fda1dfd961d7a81bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:33.142151 14438 tablet_service.cc:1467] Processing CreateTablet for tablet b0da7cee023c4066b112aae12879949f (DEFAULT_TABLE table=test-table [id=ee725fa1cef340fda1dfd961d7a81bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:33.142417 14438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0da7cee023c4066b112aae12879949f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:33.142418 14572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0da7cee023c4066b112aae12879949f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:33.143565 14304 tablet_service.cc:1467] Processing CreateTablet for tablet b0da7cee023c4066b112aae12879949f (DEFAULT_TABLE table=test-table [id=ee725fa1cef340fda1dfd961d7a81bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:47:33.143833 14304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0da7cee023c4066b112aae12879949f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:47:33.145522 14652 tablet_bootstrap.cc:492] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: Bootstrap starting.
I20260928 17:47:33.145718 14653 tablet_bootstrap.cc:492] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: Bootstrap starting.
I20260928 17:47:33.146368 14653 tablet_bootstrap.cc:654] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:33.146598 14654 tablet_bootstrap.cc:492] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675: Bootstrap starting.
I20260928 17:47:33.146862 14653 log.cc:824] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:33.147152 14654 tablet_bootstrap.cc:654] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:33.147553 14652 tablet_bootstrap.cc:654] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: Neither blocks nor log segments found. Creating new log.
I20260928 17:47:33.147679 14653 tablet_bootstrap.cc:492] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: No bootstrap required, opened a new log
I20260928 17:47:33.147698 14654 log.cc:824] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:33.147747 14653 ts_tablet_manager.cc:1397] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:33.148085 14652 log.cc:824] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: Log is configured to *not* fsync() on all Append() calls
I20260928 17:47:33.148497 14654 tablet_bootstrap.cc:492] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675: No bootstrap required, opened a new log
I20260928 17:47:33.148563 14654 ts_tablet_manager.cc:1397] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:47:33.148756 14652 tablet_bootstrap.cc:492] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: No bootstrap required, opened a new log
I20260928 17:47:33.148809 14652 ts_tablet_manager.cc:1397] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 17:47:33.149523 14653 raft_consensus.cc:348] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } }
I20260928 17:47:33.149680 14653 raft_consensus.cc:374] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:33.149712 14653 raft_consensus.cc:729] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91dafdc1d2984394b0eb2e27ba164024, State: Initialized, Role: FOLLOWER
I20260928 17:47:33.149840 14653 consensus_queue.cc:260] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } }
I20260928 17:47:33.150084 14653 ts_tablet_manager.cc:1428] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:47:33.150199 14638 heartbeater.cc:503] Master 127.3.222.62:45067 was elected leader, sending a full tablet report...
I20260928 17:47:33.150197 14652 raft_consensus.cc:348] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } }
I20260928 17:47:33.150344 14652 raft_consensus.cc:374] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:33.150381 14652 raft_consensus.cc:729] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c903f0a0ad49d493049bdb8fc98c11, State: Initialized, Role: FOLLOWER
I20260928 17:47:33.150367 14654 raft_consensus.cc:348] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } }
I20260928 17:47:33.150502 14654 raft_consensus.cc:374] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:47:33.150480 14652 consensus_queue.cc:260] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } }
I20260928 17:47:33.150537 14654 raft_consensus.cc:729] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a1c9ee8ef4a1190c6882b05da5675, State: Initialized, Role: FOLLOWER
I20260928 17:47:33.150635 14654 consensus_queue.cc:260] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } }
I20260928 17:47:33.150763 14504 heartbeater.cc:503] Master 127.3.222.62:45067 was elected leader, sending a full tablet report...
I20260928 17:47:33.150888 14370 heartbeater.cc:503] Master 127.3.222.62:45067 was elected leader, sending a full tablet report...
I20260928 17:47:33.150707 14652 ts_tablet_manager.cc:1428] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:47:33.150876 14654 ts_tablet_manager.cc:1428] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260928 17:47:33.191417 14505 tablet.cc:2369] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:47:33.233124 14660 raft_consensus.cc:482] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:47:33.233240 14660 raft_consensus.cc:504] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } }
I20260928 17:47:33.233629 14660 leader_election.cc:288] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91dafdc1d2984394b0eb2e27ba164024 (127.3.222.3:43537), 29c903f0a0ad49d493049bdb8fc98c11 (127.3.222.2:41117)
I20260928 17:47:33.237350 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.233896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:45899 (local address 127.3.222.2:41117)
0928 17:47:33.233940 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:33.233942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:33.233952 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:47:33.234033 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:33.234034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:33.234043 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:33.234089 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:33.234250 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.234990 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.236453 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.237034 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.237094 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.237105 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:33.237246 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 17:47:33.237282 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:47:33.237365 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.233918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:33167 (local address 127.3.222.3:43537)
0928 17:47:33.233959 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:33.233961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:33.234078 (+ 117us) server_negotiation.cc:407] Connection header received
0928 17:47:33.234127 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:33.234128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:33.234139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:33.234178 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:47:33.234303 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.234990 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.236454 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.237024 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.237094 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.237104 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:33.237246 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 17:47:33.237282 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:47:33.237373 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.233806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:41117 (local address 127.3.222.1:45899)
0928 17:47:33.233860 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:33.233874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:33.233944 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:33.234109 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:33.234110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:33.234111 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:33.234183 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.234185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.235073 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.235074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.236369 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.236372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.237101 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.237102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.237182 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.237184 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:47:33.237237 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 17:47:33.237285 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":71}
I20260928 17:47:33.237380 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.233826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:43537 (local address 127.3.222.1:33167)
0928 17:47:33.234011 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:33.234020 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:47:33.234072 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:47:33.234194 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:47:33.234195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:47:33.234196 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:47:33.234248 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.234249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.235073 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.235074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.236369 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:47:33.236372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.237102 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.237102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:47:33.237178 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.237182 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:47:33.237237 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 17:47:33.237285 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":33,"thread_start_us":123,"threads_started":1}
I20260928 17:47:33.237810 14458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da7cee023c4066b112aae12879949f" candidate_uuid: "957a1c9ee8ef4a1190c6882b05da5675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c903f0a0ad49d493049bdb8fc98c11" is_pre_election: true
I20260928 17:47:33.237815 14592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da7cee023c4066b112aae12879949f" candidate_uuid: "957a1c9ee8ef4a1190c6882b05da5675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91dafdc1d2984394b0eb2e27ba164024" is_pre_election: true
I20260928 17:47:33.237946 14592 raft_consensus.cc:2454] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a1c9ee8ef4a1190c6882b05da5675 in term 0.
I20260928 17:47:33.237953 14458 raft_consensus.cc:2454] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a1c9ee8ef4a1190c6882b05da5675 in term 0.
I20260928 17:47:33.238175 14258 leader_election.cc:302] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c903f0a0ad49d493049bdb8fc98c11, 957a1c9ee8ef4a1190c6882b05da5675; no voters:
I20260928 17:47:33.238288 14660 raft_consensus.cc:2781] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:47:33.238365 14660 raft_consensus.cc:482] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:47:33.238399 14660 raft_consensus.cc:3037] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:33.239101 14660 raft_consensus.cc:504] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } }
I20260928 17:47:33.239277 14660 leader_election.cc:288] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [CANDIDATE]: Term 1 election: Requested vote from peers 91dafdc1d2984394b0eb2e27ba164024 (127.3.222.3:43537), 29c903f0a0ad49d493049bdb8fc98c11 (127.3.222.2:41117)
I20260928 17:47:33.239346 14592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da7cee023c4066b112aae12879949f" candidate_uuid: "957a1c9ee8ef4a1190c6882b05da5675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91dafdc1d2984394b0eb2e27ba164024"
I20260928 17:47:33.239403 14458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da7cee023c4066b112aae12879949f" candidate_uuid: "957a1c9ee8ef4a1190c6882b05da5675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c903f0a0ad49d493049bdb8fc98c11"
I20260928 17:47:33.239426 14592 raft_consensus.cc:3037] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:33.239465 14458 raft_consensus.cc:3037] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:47:33.240051 14458 raft_consensus.cc:2454] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a1c9ee8ef4a1190c6882b05da5675 in term 1.
I20260928 17:47:33.240070 14592 raft_consensus.cc:2454] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a1c9ee8ef4a1190c6882b05da5675 in term 1.
I20260928 17:47:33.240224 14257 leader_election.cc:302] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91dafdc1d2984394b0eb2e27ba164024, 957a1c9ee8ef4a1190c6882b05da5675; no voters:
I20260928 17:47:33.240329 14660 raft_consensus.cc:2781] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:47:33.240479 14660 raft_consensus.cc:686] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 1 LEADER]: Becoming Leader. State: Replica: 957a1c9ee8ef4a1190c6882b05da5675, State: Running, Role: LEADER
I20260928 17:47:33.240562 14660 consensus_queue.cc:237] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } }
I20260928 17:47:33.241315 14178 catalog_manager.cc:5495] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a1c9ee8ef4a1190c6882b05da5675 (127.3.222.1). New cstate: current_term: 1 leader_uuid: "957a1c9ee8ef4a1190c6882b05da5675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a1c9ee8ef4a1190c6882b05da5675" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 38161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260928 17:47:33.254297 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.252130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.3.222.1:38161)
0928 17:47:33.252170 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:47:33.252172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:47:33.252183 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:47:33.252254 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:47:33.252254 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:47:33.252273 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:47:33.252334 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:47:33.252456 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.252913 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.253607 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:47:33.253800 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:47:33.253851 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:47:33.254007 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:47:33.254106 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:47:33.254173 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:47:33.254208 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 17:47:33.254227 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:47:33.255890 14458 raft_consensus.cc:1261] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [term 1 FOLLOWER]: Refusing update from remote peer 957a1c9ee8ef4a1190c6882b05da5675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:33.255889 14592 raft_consensus.cc:1261] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [term 1 FOLLOWER]: Refusing update from remote peer 957a1c9ee8ef4a1190c6882b05da5675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:47:33.256208 14662 consensus_queue.cc:1035] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91dafdc1d2984394b0eb2e27ba164024" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:33.256321 14660 consensus_queue.cc:1035] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c903f0a0ad49d493049bdb8fc98c11" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:47:33.258414 14665 mvcc.cc:205] Tried to move back new op lower bound from 7334369907734745088 to 7334369907674845184. Current Snapshot: MvccSnapshot[applied={T|T < 7334369907734745088}]
I20260928 17:47:33.258859 14667 mvcc.cc:205] Tried to move back new op lower bound from 7334369907734745088 to 7334369907674845184. Current Snapshot: MvccSnapshot[applied={T|T < 7334369907734745088}]
I20260928 17:47:33.258859 14666 mvcc.cc:205] Tried to move back new op lower bound from 7334369907734745088 to 7334369907674845184. Current Snapshot: MvccSnapshot[applied={T|T < 7334369907734745088}]
I20260928 17:47:33.260561 14179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:33.260658 14179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:47:33.261319 14179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a77129fcbbee4902b9b81e11944bc2a7: Sending DeleteTablet for 3 replicas of tablet b0da7cee023c4066b112aae12879949f
I20260928 17:47:33.261667 14572 tablet_service.cc:1514] Processing DeleteTablet for tablet b0da7cee023c4066b112aae12879949f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:33 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:57846
I20260928 17:47:33.261670 14304 tablet_service.cc:1514] Processing DeleteTablet for tablet b0da7cee023c4066b112aae12879949f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:33 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:48278
I20260928 17:47:33.261684 14438 tablet_service.cc:1514] Processing DeleteTablet for tablet b0da7cee023c4066b112aae12879949f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:47:33 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:45350
I20260928 17:47:33.261894 14677 tablet_replica.cc:330] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675: stopping tablet replica
I20260928 17:47:33.261982 14677 raft_consensus.cc:2229] T b0da7cee023c4066b112aae12879949f P 957a1c9ee8ef4a1190c6882b05da5675 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:47:33.262102 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 14241
I20260928 17:47:33.262629 14676 tablet_replica.cc:330] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: stopping tablet replica
I20260928 17:47:33.262712 14676 raft_consensus.cc:2229] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:33.262795 14676 raft_consensus.cc:2258] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:33.262948 14675 tablet_replica.cc:330] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: stopping tablet replica
I20260928 17:47:33.263021 14675 raft_consensus.cc:2229] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:47:33.263099 14675 raft_consensus.cc:2258] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:47:33.263459 14675 ts_tablet_manager.cc:1905] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:33.264910 14675 ts_tablet_manager.cc:1918] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:33.264972 14675 log.cc:1196] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/b0da7cee023c4066b112aae12879949f
I20260928 17:47:33.265179 14675 ts_tablet_manager.cc:1939] T b0da7cee023c4066b112aae12879949f P 29c903f0a0ad49d493049bdb8fc98c11: Deleting consensus metadata
I20260928 17:47:33.265493 14164 catalog_manager.cc:4841] TS 29c903f0a0ad49d493049bdb8fc98c11 (127.3.222.2:41117): tablet b0da7cee023c4066b112aae12879949f (table test-table [id=ee725fa1cef340fda1dfd961d7a81bb1]) successfully deleted
W20260928 17:47:33.267378 14163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:38161 (error 108)
I20260928 17:47:33.267560 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 14375
I20260928 17:47:33.267746 14676 ts_tablet_manager.cc:1905] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:47:33.269348 14676 ts_tablet_manager.cc:1918] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:47:33.269411 14676 log.cc:1196] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/b0da7cee023c4066b112aae12879949f
I20260928 17:47:33.269649 14676 ts_tablet_manager.cc:1939] T b0da7cee023c4066b112aae12879949f P 91dafdc1d2984394b0eb2e27ba164024: Deleting consensus metadata
I20260928 17:47:33.270004 14163 catalog_manager.cc:4841] TS 91dafdc1d2984394b0eb2e27ba164024 (127.3.222.3:43537): tablet b0da7cee023c4066b112aae12879949f (table test-table [id=ee725fa1cef340fda1dfd961d7a81bb1]) successfully deleted
I20260928 17:47:33.271593 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:47:33.271475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:38161 (local address 127.0.0.1:48292)
0928 17:47:33.271527 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 17:47:33.271566 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 17:47:33.271710 14163 catalog_manager.cc:4584] TS 957a1c9ee8ef4a1190c6882b05da5675 (127.3.222.1:38161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0da7cee023c4066b112aae12879949f: Network error: Client connection negotiation failed: client connection to 127.3.222.1:38161: connect: Connection refused (error 111)
I20260928 17:47:33.272944 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 14509
I20260928 17:47:33.276768 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 14147
2026-09-28T17:47:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1853 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:47:33 dist-test-slave-w3bm krb5kdc[14681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:47:33 dist-test-slave-w3bm krb5kdc[14681](info): set up 2 sockets
Sep 28 17:47:33 dist-test-slave-w3bm krb5kdc[14681](info): commencing operation
krb5kdc: starting...
W20260928 17:47:34.223773 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:47:34 dist-test-slave-w3bm krb5kdc[14681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:47:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/rangeradmin_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/rangeradmin_127.3.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/rangerlookup_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/rangerlookup_127.3.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/HTTP_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/HTTP_127.3.222.20@KRBTEST.COM.keytab.
I20260928 17:47:34.384178 3960 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkw9tqR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/postgres -l logfile start
2026-09-28 17:47:36.786 UTC [14711] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 17:47:36.786 UTC [14711] LOG: listening on IPv4 address "127.3.222.20", port 53267
2026-09-28 17:47:36.791 UTC [14711] LOG: listening on Unix socket "/tmp/.s.PGSQL.53267"
2026-09-28 17:47:36.816 UTC [14714] LOG: database system was shut down at 2026-09-28 17:47:35 UTC
2026-09-28 17:47:36.820 UTC [14711] LOG: database system is ready to accept connections
I20260928 17:47:37.679745 3960 mini_postgres.cc:96] Postgres bound to 53267
2026-09-28 17:47:37.685 UTC [14722] FATAL: database "slave" does not exist
127.3.222.20:53267 - accepting connections
I20260928 17:47:37.685715 3960 mini_ranger.cc:161] Starting Ranger...
I20260928 17:47:37.699302 3960 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 17:47:38.168879 3960 mini_ranger.cc:87] Created ranger Postgres database
I20260928 17:47:38.168953 3960 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-admin
2026-09-28 17:47:38,426 [I] DB FLAVOR :POSTGRES
2026-09-28 17:47:38,427 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 17:47:38,427 [I] Checking connection..
2026-09-28 17:47:38,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 17:47:38,666 [I] Checking connection passed.
2026-09-28 17:47:38,667 [I] --------- Verifying version history table ---------
2026-09-28 17:47:38,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:47:38,914 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 17:47:38,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:47:39,161 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 17:47:39,162 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 17:47:39,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 17:47:39.405 UTC [14786] WARNING: there is no transaction in progress
2026-09-28 17:47:39,410 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 17:47:39,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:47:39,632 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 17:47:39,632 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 17:47:39,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 17:47:39,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 17:47:40,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w3bm.c.gcp-upstream.internal','N') ;"
2026-09-28 17:47:40,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 17:47:40,587 [I] Table x_portal_user does not exist in database ranger
2026-09-28 17:47:40,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 17:47:40,835 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 17:47:40,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 17:47:41,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 17:47:41,330 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 17:47:41,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 17:47:42.273 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.290 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.305 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.320 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.336 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.354 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.408 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.414 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.428 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.437 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.448 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.461 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.471 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.478 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.485 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:42.492 UTC [14894] WARNING: there is no transaction in progress
2026-09-28 17:47:43,138 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 17:47:43,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 17:47:43,374 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 17:47:43,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 17:47:43,609 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 17:47:43,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 17:47:43,837 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 17:47:43,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 17:47:44,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 17:47:44,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w3bm.c.gcp-upstream.internal';"
2026-09-28 17:47:44,578 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 17:47:44,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 17:47:44,817 [I] DB_PATCHES have already been applied
I20260928 17:47:44.822860 3960 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkw9tqR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 17:47:44.822923 3960 mini_ranger.cc:193] Using host: 127.3.222.20
I20260928 17:47:44.825645 3960 mini_ranger.cc:227] Ranger admin URL: http://127.3.222.20:58311
Sep 28, 2026 5:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 17:47:45.792902 3960 mini_ranger.cc:160] Time spent starting Ranger: real 8.107s user 0.000s sys 0.007s
Sep 28, 2026 5:48:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:48:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 5:48:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:48:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:48:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 17:48:05.821328 3960 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/rangerkms_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/rangerkms_127.3.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.222.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.222.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.222.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/HTTP_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.222.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/HTTP_127.3.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 17:48:05 dist-test-slave-w3bm krb5kdc[14681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617685, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 17:48:05.905349 3960 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 17:48:05.914047 3960 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 17:48:06.771572 3960 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 17:48:06.771652 3960 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-kms
I20260928 17:48:06.771668 3960 mini_ranger_kms.cc:227] Using postgres at 127.3.222.20:53267
2026-09-28 17:48:06,930 [I] DB FLAVOR :POSTGRES
2026-09-28 17:48:06,930 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 17:48:06,930 [I] Checking connection
2026-09-28 17:48:06,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 17:48:07,183 [I] connection success
2026-09-28 17:48:07,183 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 17:48:07,183 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 17:48:07,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 17:48:07,428 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 17:48:07,429 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 17:48:07,429 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 17:48:07,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:53267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 17:48:07,691 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 17:48:08.108459 3960 mini_ranger_kms.cc:310] Created kms service
I20260928 17:48:08.274984 3960 mini_ranger_kms.cc:326] Created kudu user
I20260928 17:48:08.316169 3960 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 17:48:08 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617685, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:08.680209 3960 mini_ranger_kms.cc:384] Added ranger policy
I20260928 17:48:08.680328 3960 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 17:48:08.680348 3960 mini_ranger_kms.cc:240] Using host: 127.3.222.20
I20260928 17:48:08.683483 3960 mini_ranger_kms.cc:276] Ranger KMS PID: 15074
I20260928 17:48:08.683593 3960 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.222.20:50507
Sep 28, 2026 5:48:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:48:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:48:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:48:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:48:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:48:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 17:48:10.715008 3960 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.810s user 0.007s sys 0.136s
I20260928 17:48:10.715124 3960 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 17:48:10.715200 3960 mini_ranger_kms.cc:401] 127.3.222.20:50507/kms/v1/keys
Sep 28 17:48:10 dist-test-slave-w3bm krb5kdc[14681](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790617690, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.222.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 17:48:11 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617690, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.222.20@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
Sep 28 17:48:11 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617690, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.222.20@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
Sep 28 17:48:13 dist-test-slave-w3bm krb5kdc[14681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:48:13.877821 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:40371
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:40371
--ranger_config_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:50507/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:48:13.956648 15125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:13.956851 15125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:13.956879 15125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:13.958031 15125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:48:13.958122 15125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:13.958184 15125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:13.958214 15125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:48:13.958235 15125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:48:13.960196 15125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.222.20:50507/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:40371
--ranger_config_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:40371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:13.960445 15125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:13.960638 15125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:13.965608 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:13.965642 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:13.965593 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:13.966251 15125 server_base.cc:1034] running on GCE node
I20260928 17:48:13.966487 15125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:13.966753 15125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:13.967892 15125 hybrid_clock.cc:648] HybridClock initialized: now 1790617693967882 us; error 28 us; skew 500 ppm
Sep 28 17:48:13 dist-test-slave-w3bm krb5kdc[14681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:13.970211 15125 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:48:13.970628 15125 webserver.cc:460] Webserver started at http://127.3.222.62:44797/ using document root <none> and password file <none>
I20260928 17:48:13.970831 15125 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:13.970868 15125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:48:13.970975 15125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:48:13 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:14.021209 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "24544f3089644d6fa60b9d12fd6e5d15"
format_stamp: "Formatted at 2026-09-28 17:48:14 on dist-test-slave-w3bm"
server_key: "3639d8979ad3d9a6c195ac1fa208d4e5"
server_key_iv: "099d23d8952c36e24a243336eb6184fa"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:14.021589 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "24544f3089644d6fa60b9d12fd6e5d15"
format_stamp: "Formatted at 2026-09-28 17:48:14 on dist-test-slave-w3bm"
server_key: "3639d8979ad3d9a6c195ac1fa208d4e5"
server_key_iv: "099d23d8952c36e24a243336eb6184fa"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:14.022956 15125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 17:48:14.033810 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:14.033950 15125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:48:14.034044 15125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "24544f3089644d6fa60b9d12fd6e5d15"
format_stamp: "Formatted at 2026-09-28 17:48:14 on dist-test-slave-w3bm"
server_key: "3639d8979ad3d9a6c195ac1fa208d4e5"
server_key_iv: "099d23d8952c36e24a243336eb6184fa"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:14.034132 15125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:14.050380 15125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:14.052102 15125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:14.052225 15125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:14.056871 15125 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:40371
I20260928 17:48:14.056892 15192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:40371 every 8 connection(s)
I20260928 17:48:14.057230 15125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:48:14.057847 15193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:48:14.060039 15193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15: Bootstrap starting.
I20260928 17:48:14.060603 15193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15: Neither blocks nor log segments found. Creating new log.
I20260928 17:48:14.060847 15193 log.cc:824] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15: Log is configured to *not* fsync() on all Append() calls
I20260928 17:48:14.061429 15193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15: No bootstrap required, opened a new log
I20260928 17:48:14.063174 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 15125
I20260928 17:48:14.063295 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:48:14.063490 3960 external_mini_cluster.cc:1428] Setting key 1c13f2bdb0f9f38cebbf86358822fecf
I20260928 17:48:14.063418 15193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24544f3089644d6fa60b9d12fd6e5d15" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40371 } }
I20260928 17:48:14.063560 15193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:48:14.063581 15193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24544f3089644d6fa60b9d12fd6e5d15, State: Initialized, Role: FOLLOWER
I20260928 17:48:14.063723 15193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24544f3089644d6fa60b9d12fd6e5d15" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40371 } }
I20260928 17:48:14.064045 15193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:48:14.064149 15193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:48:14.064242 15193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:48:14.064843 15193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24544f3089644d6fa60b9d12fd6e5d15" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40371 } }
I20260928 17:48:14.064939 15193 leader_election.cc:302] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24544f3089644d6fa60b9d12fd6e5d15; no voters:
I20260928 17:48:14.065093 15193 leader_election.cc:288] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:48:14.065397 15193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:48:14.065126 15198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:48:14.066787 15198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [term 1 LEADER]: Becoming Leader. State: Replica: 24544f3089644d6fa60b9d12fd6e5d15, State: Running, Role: LEADER
I20260928 17:48:14.066944 15198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24544f3089644d6fa60b9d12fd6e5d15" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40371 } }
I20260928 17:48:14.067595 15199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24544f3089644d6fa60b9d12fd6e5d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24544f3089644d6fa60b9d12fd6e5d15" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40371 } } }
I20260928 17:48:14.067672 15199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [sys.catalog]: This master's current role is: LEADER
I20260928 17:48:14.068498 15213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:48:14.068742 15193 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 17:48:14.069046 15205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24544f3089644d6fa60b9d12fd6e5d15. Latest consensus state: current_term: 1 leader_uuid: "24544f3089644d6fa60b9d12fd6e5d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24544f3089644d6fa60b9d12fd6e5d15" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40371 } } }
I20260928 17:48:14.069118 15205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15 [sys.catalog]: This master's current role is: LEADER
I20260928 17:48:14.069036 15213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 17:48:14 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:48:14.075131 15213 catalog_manager.cc:1347] Generated new cluster ID: 35fe26b3bc384b3793d33908943e6fff
I20260928 17:48:14.075241 15213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:48:14.084692 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:14.063922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.3.222.62:40371)
0928 17:48:14.064074 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:14.064083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:14.064161 (+ 78us) server_negotiation.cc:407] Connection header received
0928 17:48:14.064286 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:14.064294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:14.064350 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:14.064489 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:14.065200 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:14.066536 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:14.067815 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:14.068253 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:14.068342 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:14.071292 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:14.071330 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:14.071330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:14.071415 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:14.074677 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:14.075108 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:14.075114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:14.075115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:14.083404 (+ 8289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:14.083680 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:14.083681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:14.083682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:14.084024 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:14.084175 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:14.084312 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 17:48:14.084462 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260928 17:48:14.086431 15213 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:48:14.087169 15213 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:48:14.092401 15213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15: Generated new TSK 0
I20260928 17:48:14.092630 15213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:48:14.801908 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:50507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:40371
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:48:14.902956 15239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:14.903201 15239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:14.903251 15239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:14.904386 15239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:14.904426 15239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:14.904460 15239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:48:14.905908 15239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.222.20:50507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:40371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:14.906136 15239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:14.906350 15239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:14.912202 15244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:14.912247 15245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:14.912213 15247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:14.912751 15239 server_base.cc:1034] running on GCE node
I20260928 17:48:14.912928 15239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:14.913151 15239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:14.914294 15239 hybrid_clock.cc:648] HybridClock initialized: now 1790617694914281 us; error 29 us; skew 500 ppm
Sep 28 17:48:14 dist-test-slave-w3bm krb5kdc[14681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617694, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:14.916771 15239 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:48:14.917155 15239 webserver.cc:460] Webserver started at http://127.3.222.1:36795/ using document root <none> and password file <none>
I20260928 17:48:14.917353 15239 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:14.917399 15239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:48:14.917511 15239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:48:14 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617694, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:14.929665 15239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "54a1fb5eaab447eea773a364a79d0072"
format_stamp: "Formatted at 2026-09-28 17:48:14 on dist-test-slave-w3bm"
server_key: "9c7550d4929fbdf0b10fa9c625f9fb57"
server_key_iv: "de2c5dd11e9315f8255a6edcd88d4264"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:14.930106 15239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "54a1fb5eaab447eea773a364a79d0072"
format_stamp: "Formatted at 2026-09-28 17:48:14 on dist-test-slave-w3bm"
server_key: "9c7550d4929fbdf0b10fa9c625f9fb57"
server_key_iv: "de2c5dd11e9315f8255a6edcd88d4264"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:14.931875 15239 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 17:48:14.952526 15256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:14.952801 15239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:48:14.952895 15239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "54a1fb5eaab447eea773a364a79d0072"
format_stamp: "Formatted at 2026-09-28 17:48:14 on dist-test-slave-w3bm"
server_key: "9c7550d4929fbdf0b10fa9c625f9fb57"
server_key_iv: "de2c5dd11e9315f8255a6edcd88d4264"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:14.953001 15239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:14.970474 15239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:14.972782 15239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:14.972956 15239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:14.973223 15239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:48:14.973577 15239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:48:14.973644 15239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:14.973695 15239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:48:14.973735 15239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:14.981566 15239 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:44025
I20260928 17:48:14.982005 15239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:48:14.982599 15369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:44025 every 8 connection(s)
I20260928 17:48:14.987988 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 15239
I20260928 17:48:14.988101 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:48:14.988287 3960 external_mini_cluster.cc:1428] Setting key b65f7afeb8b597da9b2583ec0fd3d17d
Sep 28 17:48:14 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617694, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:48:14.992741 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:14.982744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40371 (local address 127.3.222.1:45507)
0928 17:48:14.982893 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:14.982935 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:14.983571 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:14.983967 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:14.983971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:14.984207 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:48:14.984467 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:14.984472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:14.985479 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:14.985480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:14.986688 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:14.986691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:14.987201 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:14.987202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:14.987366 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:14.987900 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:48:14.987909 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:48:14.989451 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:48:14.991198 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:14.991229 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:14.991231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:14.991499 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:14.991710 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:14.991710 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:14.991711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:14.991829 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:14.992108 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:48:14.992113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:48:14.992284 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 17:48:14.992376 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 17:48:14.992516 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260928 17:48:14.993340 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:14.983020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:45507 (local address 127.3.222.62:40371)
0928 17:48:14.983749 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:14.983751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:14.983758 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:48:14.983784 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:14.983785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:14.983796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:14.983849 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:14.984568 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:14.985378 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:14.986777 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:14.987076 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:14.987133 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:14.989543 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:14.989544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:14.989545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:14.989551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:14.991113 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:14.991579 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:14.991580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:14.991580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:14.991646 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:14.991892 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:14.991892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:14.991893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:14.992017 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:14.992084 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:14.993168 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0928 17:48:14.993250 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":72,"threads_started":1}
I20260928 17:48:14.993819 15370 heartbeater.cc:347] Connected to a master server at 127.3.222.62:40371
I20260928 17:48:14.993908 15370 heartbeater.cc:464] Registering TS with master...
I20260928 17:48:14.994073 15370 heartbeater.cc:511] Master 127.3.222.62:40371 requested a full tablet report, sending...
I20260928 17:48:14.994586 15157 ts_manager.cc:194] Registered new tserver with Master: 54a1fb5eaab447eea773a364a79d0072 (127.3.222.1:44025)
I20260928 17:48:14.995522 15157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:45507
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:48:15.034339 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:50507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:40371
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:48:15 dist-test-slave-w3bm krb5kdc[14681](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790617695, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 17:48:15.176491 15376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:15.176813 15376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:15.176896 15376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:15.178565 15376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:15.178668 15376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:15.178745 15376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:48:15.180841 15376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.222.20:50507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:40371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:15.181358 15376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:15.181675 15376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:15.187911 15384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:15.188218 15383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:15.188467 15386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:15.190054 15376 server_base.cc:1034] running on GCE node
I20260928 17:48:15.190266 15376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:15.190534 15376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:15.199272 15376 hybrid_clock.cc:648] HybridClock initialized: now 1790617695197938 us; error 1352 us; skew 500 ppm
Sep 28 17:48:15 dist-test-slave-w3bm krb5kdc[14681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617695, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:15.202618 15376 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:48:15.203150 15376 webserver.cc:460] Webserver started at http://127.3.222.2:34569/ using document root <none> and password file <none>
I20260928 17:48:15.203433 15376 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:15.203485 15376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:48:15.203606 15376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:48:15 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617695, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:15.226025 15376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "151fc31043af47da9b8fb75ebd5413dd"
format_stamp: "Formatted at 2026-09-28 17:48:15 on dist-test-slave-w3bm"
server_key: "89a0e436b1cc48a958dd2b0c4f5e2eeb"
server_key_iv: "ff8740757e142a4d70d0772c64baf6a3"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:15.226405 15376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "151fc31043af47da9b8fb75ebd5413dd"
format_stamp: "Formatted at 2026-09-28 17:48:15 on dist-test-slave-w3bm"
server_key: "89a0e436b1cc48a958dd2b0c4f5e2eeb"
server_key_iv: "ff8740757e142a4d70d0772c64baf6a3"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:15.228266 15376 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 17:48:15.241667 15393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:15.241858 15376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:48:15.241945 15376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "151fc31043af47da9b8fb75ebd5413dd"
format_stamp: "Formatted at 2026-09-28 17:48:15 on dist-test-slave-w3bm"
server_key: "89a0e436b1cc48a958dd2b0c4f5e2eeb"
server_key_iv: "ff8740757e142a4d70d0772c64baf6a3"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:15.242039 15376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:15.279659 15376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:15.282527 15376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:15.282711 15376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:15.283002 15376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:48:15.283455 15376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:48:15.283545 15376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:15.283614 15376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:48:15.283670 15376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:15.292285 15376 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:36219
I20260928 17:48:15.292748 15376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:48:15.294273 15507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:36219 every 8 connection(s)
Sep 28 17:48:15 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617695, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:48:15.301606 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 15376
I20260928 17:48:15.301707 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:48:15.301900 3960 external_mini_cluster.cc:1428] Setting key a38ace1c9be6628372f70126657404c1
I20260928 17:48:15.303769 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.295099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:49463 (local address 127.3.222.62:40371)
0928 17:48:15.295148 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:15.295150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:15.295544 (+ 394us) server_negotiation.cc:407] Connection header received
0928 17:48:15.296123 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:15.296124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:15.296139 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:15.296190 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:15.296713 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.297219 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.298158 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.298379 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.298437 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.300298 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:15.300300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:15.300300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:15.300307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:15.301583 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:15.302031 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:15.302032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:15.302032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:15.302089 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:15.302341 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:15.302342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:15.302342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:15.302472 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:15.302528 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:15.303598 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0928 17:48:15.303680 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:48:15.304390 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.294821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40371 (local address 127.3.222.2:49463)
0928 17:48:15.295292 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:15.295450 (+ 158us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:15.296017 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:15.296253 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:15.296256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:15.296433 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:48:15.296623 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.296626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.297303 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.297303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.298076 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.298078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.298449 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.298450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.298558 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.299009 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:48:15.299017 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:48:15.300217 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:48:15.301670 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:15.301677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:15.301679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:15.301946 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:15.302155 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:15.302155 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:15.302155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:15.302278 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:15.302553 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:48:15.302559 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:48:15.303444 (+ 885us) client_negotiation.cc:769] Sending connection context
0928 17:48:15.304079 (+ 635us) client_negotiation.cc:240] Negotiation successful
0928 17:48:15.304221 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":75,"threads_started":1}
I20260928 17:48:15.304809 15508 heartbeater.cc:347] Connected to a master server at 127.3.222.62:40371
I20260928 17:48:15.304898 15508 heartbeater.cc:464] Registering TS with master...
I20260928 17:48:15.305063 15508 heartbeater.cc:511] Master 127.3.222.62:40371 requested a full tablet report, sending...
I20260928 17:48:15.305382 15157 ts_manager.cc:194] Registered new tserver with Master: 151fc31043af47da9b8fb75ebd5413dd (127.3.222.2:36219)
I20260928 17:48:15.305727 15157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:49463
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:48:15.332028 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:50507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:40371
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:48:15.457144 15513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:15.457540 15513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:15.457618 15513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:15.459286 15513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:15.459381 15513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:15.459434 15513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:48:15.460937 15513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.222.20:50507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:40371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:15.461355 15513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:15.461658 15513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:15.468164 15521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:15.468144 15519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:15.472277 15518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:15.474913 15513 server_base.cc:1034] running on GCE node
I20260928 17:48:15.475276 15513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:15.475555 15513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:15.483035 15513 hybrid_clock.cc:648] HybridClock initialized: now 1790617695483012 us; error 48 us; skew 500 ppm
Sep 28 17:48:15 dist-test-slave-w3bm krb5kdc[14681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617695, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:15.486135 15513 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:48:15.486549 15513 webserver.cc:460] Webserver started at http://127.3.222.3:41133/ using document root <none> and password file <none>
I20260928 17:48:15.486761 15513 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:15.486809 15513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:48:15.486953 15513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:48:15 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617695, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:15.499954 15513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "f42cf4debec64ee184a375be54256728"
format_stamp: "Formatted at 2026-09-28 17:48:15 on dist-test-slave-w3bm"
server_key: "4b6de28668910c00b4b0757776616854"
server_key_iv: "0157530d0ef43a38f26a49bfa645a5b6"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:15.500358 15513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "f42cf4debec64ee184a375be54256728"
format_stamp: "Formatted at 2026-09-28 17:48:15 on dist-test-slave-w3bm"
server_key: "4b6de28668910c00b4b0757776616854"
server_key_iv: "0157530d0ef43a38f26a49bfa645a5b6"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:15.508598 15513 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
I20260928 17:48:15.529512 15528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:15.529770 15513 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260928 17:48:15.529857 15513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "f42cf4debec64ee184a375be54256728"
format_stamp: "Formatted at 2026-09-28 17:48:15 on dist-test-slave-w3bm"
server_key: "4b6de28668910c00b4b0757776616854"
server_key_iv: "0157530d0ef43a38f26a49bfa645a5b6"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:15.529949 15513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:15.557934 15513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:15.560359 15513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:15.560554 15513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:15.560822 15513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:48:15.561237 15513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:48:15.561334 15513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:15.561410 15513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:48:15.561466 15513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:15.570906 15513 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:45627
I20260928 17:48:15.571906 15513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:48:15.574290 15641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:45627 every 8 connection(s)
I20260928 17:48:15.575287 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 15513
I20260928 17:48:15.575376 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:48:15.575560 3960 external_mini_cluster.cc:1428] Setting key 6147c8ac42bb262a9e9a5f5d5c4b427e
Sep 28 17:48:15 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617695, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:48:15.589823 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.575018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:46461 (local address 127.3.222.62:40371)
0928 17:48:15.575077 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:15.575080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:15.579667 (+ 4587us) server_negotiation.cc:407] Connection header received
0928 17:48:15.580320 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:15.580322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:15.580333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:15.580362 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:15.581214 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.581986 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.583551 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.583829 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.583870 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.586511 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:15.586512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:15.586513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:15.586520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:15.588128 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:15.588584 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:15.588585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:15.588585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:15.588643 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:15.588915 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:15.588916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:15.588916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:15.589046 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:15.589100 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:15.589640 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 17:48:15.589726 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 17:48:15.590682 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.575012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:40371 (local address 127.3.222.3:46461)
0928 17:48:15.579003 (+ 3991us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:15.579055 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:15.580199 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:15.580553 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:15.580557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:15.580797 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:48:15.581098 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.581105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.582115 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.582116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.583450 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.583453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.583960 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.583961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.584115 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.584704 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:48:15.584714 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:48:15.586404 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:48:15.588219 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:15.588225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:15.588228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:15.588501 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:15.588711 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:15.588712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:15.588712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:15.588842 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:15.589145 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:48:15.589151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:48:15.589328 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 17:48:15.590288 (+ 960us) client_negotiation.cc:240] Negotiation successful
0928 17:48:15.590451 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3943,"thread_start_us":77,"threads_started":1}
I20260928 17:48:15.591338 15642 heartbeater.cc:347] Connected to a master server at 127.3.222.62:40371
I20260928 17:48:15.591431 15642 heartbeater.cc:464] Registering TS with master...
I20260928 17:48:15.591611 15642 heartbeater.cc:511] Master 127.3.222.62:40371 requested a full tablet report, sending...
I20260928 17:48:15.592011 15157 ts_manager.cc:194] Registered new tserver with Master: f42cf4debec64ee184a375be54256728 (127.3.222.3:45627)
I20260928 17:48:15.592507 15157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:46461
I20260928 17:48:15.597613 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:48:15.607654 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.599382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.3.222.62:40371)
0928 17:48:15.599419 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:15.599421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:15.600274 (+ 853us) server_negotiation.cc:407] Connection header received
0928 17:48:15.600362 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:15.600363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:15.600372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:15.600398 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:15.601125 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.601831 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.602913 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.603233 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.603277 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.604808 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:15.604810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:15.604811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:15.604818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:15.606410 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:15.606789 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:15.606790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:15.606791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:15.606846 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:15.607041 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:15.607042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:15.607042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:15.607175 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:15.607280 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:15.607486 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 17:48:15.607558 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:48:15.609787 15157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:48:15.612109 15157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:48:15.620954 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.614860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:36219 (local address 127.0.0.1:43504)
0928 17:48:15.615743 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:15.615784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:15.615842 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:15.616371 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:15.616374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:15.616377 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:48:15.616442 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.616444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.618024 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.618025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.619349 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.619352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.620630 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.620631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.620714 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.620730 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 17:48:15.620821 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 17:48:15.620876 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":78,"threads_started":1}
I20260928 17:48:15.621625 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.614954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.3.222.2:36219)
0928 17:48:15.616192 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:15.616198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:15.616208 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:48:15.616246 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:15.616248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:15.616264 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:15.616509 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:48:15.616581 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.617938 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.619466 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.620532 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.621393 (+ 861us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.621483 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:15.621510 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 17:48:15.621549 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":81,"threads_started":1}
I20260928 17:48:15.622462 15442 tablet_service.cc:1467] Processing CreateTablet for tablet 7914d3d6294a482d88a08846d137ca7d (DEFAULT_TABLE table=test-table [id=def11ad3aad5471481eb40e86ae8e043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:48:15.622856 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7914d3d6294a482d88a08846d137ca7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:48:15.628058 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.615327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44025 (local address 127.0.0.1:42564)
0928 17:48:15.616027 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:15.616034 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:15.616077 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:15.622279 (+ 6202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:15.622280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:15.622283 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:48:15.622559 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.622562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.623869 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.623870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.625020 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.625023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.627833 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.627834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.627898 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.627903 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:48:15.627932 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 17:48:15.627968 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":73,"threads_started":1}
I20260928 17:48:15.628599 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.615425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.3.222.1:44025)
0928 17:48:15.622114 (+ 6689us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:15.622119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:15.622129 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:48:15.622162 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:15.622164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:15.622179 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:15.622644 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:48:15.622718 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.623768 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.625129 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.627741 (+ 2612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.628393 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.628461 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:15.628487 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 17:48:15.628528 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6652,"thread_start_us":46,"threads_started":1}
I20260928 17:48:15.629350 15304 tablet_service.cc:1467] Processing CreateTablet for tablet 7914d3d6294a482d88a08846d137ca7d (DEFAULT_TABLE table=test-table [id=def11ad3aad5471481eb40e86ae8e043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:48:15.629694 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7914d3d6294a482d88a08846d137ca7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:48:15.635804 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.615096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:45627 (local address 127.0.0.1:47962)
0928 17:48:15.615933 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:15.615940 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:15.615970 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:15.626125 (+ 10155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:15.626126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:15.626128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:48:15.626179 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.626180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.633338 (+ 7158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.633339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.634508 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.634511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.635585 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.635586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.635658 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.635663 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:48:15.635691 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 17:48:15.635726 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":70,"threads_started":1}
I20260928 17:48:15.636437 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.615192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47962 (local address 127.3.222.3:45627)
0928 17:48:15.625950 (+ 10758us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:15.625954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:15.625964 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:48:15.625999 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:15.626001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:15.626016 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:15.626241 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:48:15.626313 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.627404 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.634647 (+ 7243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.635487 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.636165 (+ 678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.636301 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:15.636328 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 17:48:15.636369 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10718,"thread_start_us":96,"threads_started":1}
I20260928 17:48:15.637182 15576 tablet_service.cc:1467] Processing CreateTablet for tablet 7914d3d6294a482d88a08846d137ca7d (DEFAULT_TABLE table=test-table [id=def11ad3aad5471481eb40e86ae8e043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:48:15.637511 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7914d3d6294a482d88a08846d137ca7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:48:15.640180 15661 tablet_bootstrap.cc:492] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072: Bootstrap starting.
I20260928 17:48:15.640910 15661 tablet_bootstrap.cc:654] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072: Neither blocks nor log segments found. Creating new log.
I20260928 17:48:15.641263 15661 log.cc:824] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072: Log is configured to *not* fsync() on all Append() calls
I20260928 17:48:15.641395 15660 tablet_bootstrap.cc:492] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: Bootstrap starting.
I20260928 17:48:15.642881 15660 tablet_bootstrap.cc:654] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: Neither blocks nor log segments found. Creating new log.
I20260928 17:48:15.643261 15660 log.cc:824] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: Log is configured to *not* fsync() on all Append() calls
I20260928 17:48:15.643615 15662 tablet_bootstrap.cc:492] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: Bootstrap starting.
I20260928 17:48:15.644577 15662 tablet_bootstrap.cc:654] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: Neither blocks nor log segments found. Creating new log.
I20260928 17:48:15.644897 15662 log.cc:824] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: Log is configured to *not* fsync() on all Append() calls
I20260928 17:48:15.646000 15660 tablet_bootstrap.cc:492] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: No bootstrap required, opened a new log
I20260928 17:48:15.646122 15660 ts_tablet_manager.cc:1397] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 17:48:15.647775 15661 tablet_bootstrap.cc:492] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072: No bootstrap required, opened a new log
I20260928 17:48:15.647907 15661 ts_tablet_manager.cc:1397] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 17:48:15.648209 15660 raft_consensus.cc:348] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } }
I20260928 17:48:15.648470 15660 raft_consensus.cc:374] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:48:15.648554 15660 raft_consensus.cc:729] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151fc31043af47da9b8fb75ebd5413dd, State: Initialized, Role: FOLLOWER
I20260928 17:48:15.648756 15660 consensus_queue.cc:260] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } }
I20260928 17:48:15.649111 15660 ts_tablet_manager.cc:1428] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 17:48:15.649295 15508 heartbeater.cc:503] Master 127.3.222.62:40371 was elected leader, sending a full tablet report...
I20260928 17:48:15.649753 15662 tablet_bootstrap.cc:492] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: No bootstrap required, opened a new log
I20260928 17:48:15.649868 15662 ts_tablet_manager.cc:1397] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 17:48:15.650434 15661 raft_consensus.cc:348] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } }
I20260928 17:48:15.650660 15661 raft_consensus.cc:374] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:48:15.650736 15661 raft_consensus.cc:729] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a1fb5eaab447eea773a364a79d0072, State: Initialized, Role: FOLLOWER
I20260928 17:48:15.650906 15661 consensus_queue.cc:260] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } }
I20260928 17:48:15.651201 15661 ts_tablet_manager.cc:1428] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:48:15.651369 15370 heartbeater.cc:503] Master 127.3.222.62:40371 was elected leader, sending a full tablet report...
I20260928 17:48:15.652493 15662 raft_consensus.cc:348] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } }
I20260928 17:48:15.652786 15662 raft_consensus.cc:374] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:48:15.652858 15662 raft_consensus.cc:729] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42cf4debec64ee184a375be54256728, State: Initialized, Role: FOLLOWER
I20260928 17:48:15.653018 15662 consensus_queue.cc:260] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } }
I20260928 17:48:15.653306 15662 ts_tablet_manager.cc:1428] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 17:48:15.653771 15642 heartbeater.cc:503] Master 127.3.222.62:40371 was elected leader, sending a full tablet report...
Sep 28 17:48:15 dist-test-slave-w3bm krb5kdc[14681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617695, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:15.729068 15666 raft_consensus.cc:482] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:48:15.729243 15666 raft_consensus.cc:504] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } }
I20260928 17:48:15.729635 15666 leader_election.cc:288] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54a1fb5eaab447eea773a364a79d0072 (127.3.222.1:44025), f42cf4debec64ee184a375be54256728 (127.3.222.3:45627)
W20260928 17:48:15.733026 15371 tablet.cc:2369] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:48:15.733558 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.729792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44025 (local address 127.3.222.2:36723)
0928 17:48:15.729836 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:15.729849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:15.729892 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:15.730289 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:15.730290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:15.730291 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:48:15.730360 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.730362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.731328 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.731329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.732531 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.732534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.733322 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.733323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.733399 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.733402 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:48:15.733448 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:48:15.733485 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 17:48:15.734202 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.729950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:36723 (local address 127.3.222.1:44025)
0928 17:48:15.730040 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:15.730042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:15.730049 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:48:15.730077 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:15.730078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:15.730089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:15.730134 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:48:15.730444 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.731240 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.732619 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.733144 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.733201 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.733212 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:15.734085 (+ 873us) server_negotiation.cc:299] Negotiation successful
0928 17:48:15.734129 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 17:48:15.734439 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7914d3d6294a482d88a08846d137ca7d" candidate_uuid: "151fc31043af47da9b8fb75ebd5413dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a1fb5eaab447eea773a364a79d0072" is_pre_election: true
I20260928 17:48:15.734593 15324 raft_consensus.cc:2454] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151fc31043af47da9b8fb75ebd5413dd in term 0.
I20260928 17:48:15.735566 15396 leader_election.cc:302] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151fc31043af47da9b8fb75ebd5413dd, 54a1fb5eaab447eea773a364a79d0072; no voters:
I20260928 17:48:15.735687 15666 raft_consensus.cc:2781] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:48:15.735725 15666 raft_consensus.cc:482] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:48:15.735743 15666 raft_consensus.cc:3037] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:48:15.736593 15666 raft_consensus.cc:504] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } }
I20260928 17:48:15.736711 15666 leader_election.cc:288] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [CANDIDATE]: Term 1 election: Requested vote from peers 54a1fb5eaab447eea773a364a79d0072 (127.3.222.1:44025), f42cf4debec64ee184a375be54256728 (127.3.222.3:45627)
I20260928 17:48:15.737015 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7914d3d6294a482d88a08846d137ca7d" candidate_uuid: "151fc31043af47da9b8fb75ebd5413dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a1fb5eaab447eea773a364a79d0072"
I20260928 17:48:15.737080 15324 raft_consensus.cc:3037] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:48:15.737869 15324 raft_consensus.cc:2454] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151fc31043af47da9b8fb75ebd5413dd in term 1.
I20260928 17:48:15.738066 15396 leader_election.cc:302] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151fc31043af47da9b8fb75ebd5413dd, 54a1fb5eaab447eea773a364a79d0072; no voters:
I20260928 17:48:15.738181 15666 raft_consensus.cc:2781] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:48:15.738309 15666 raft_consensus.cc:686] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 1 LEADER]: Becoming Leader. State: Replica: 151fc31043af47da9b8fb75ebd5413dd, State: Running, Role: LEADER
I20260928 17:48:15.738386 15666 consensus_queue.cc:237] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } }
I20260928 17:48:15.739151 15157 catalog_manager.cc:5495] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 151fc31043af47da9b8fb75ebd5413dd (127.3.222.2). New cstate: current_term: 1 leader_uuid: "151fc31043af47da9b8fb75ebd5413dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151fc31043af47da9b8fb75ebd5413dd" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 } health_report { overall_health: UNKNOWN } } }
I20260928 17:48:15.741943 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.729832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:45627 (local address 127.3.222.2:51301)
0928 17:48:15.733621 (+ 3789us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:15.733631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:15.733668 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:15.734483 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:15.734484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:15.734486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:48:15.734517 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.734518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.739630 (+ 5112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.739631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.740722 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:15.740724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.741740 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.741741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:15.741799 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.741802 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:48:15.741847 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 17:48:15.741876 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3778,"thread_start_us":102,"threads_started":1}
I20260928 17:48:15.742234 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.729999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:51301 (local address 127.3.222.3:45627)
0928 17:48:15.730200 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:15.730202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:15.733717 (+ 3515us) server_negotiation.cc:407] Connection header received
0928 17:48:15.733744 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:15.733745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:15.733755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:15.733795 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:48:15.734596 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.735326 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.741025 (+ 5699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.741575 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.741634 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.741645 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:15.742119 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 17:48:15.742173 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260928 17:48:15.742688 15589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7914d3d6294a482d88a08846d137ca7d" candidate_uuid: "151fc31043af47da9b8fb75ebd5413dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42cf4debec64ee184a375be54256728" is_pre_election: true
I20260928 17:48:15.742820 15589 raft_consensus.cc:2454] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151fc31043af47da9b8fb75ebd5413dd in term 0.
I20260928 17:48:15.742908 15594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7914d3d6294a482d88a08846d137ca7d" candidate_uuid: "151fc31043af47da9b8fb75ebd5413dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42cf4debec64ee184a375be54256728"
I20260928 17:48:15.742949 15594 raft_consensus.cc:3037] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:48:15.743729 15594 raft_consensus.cc:2454] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151fc31043af47da9b8fb75ebd5413dd in term 1.
I20260928 17:48:15.767050 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.763691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.3.222.2:36219)
0928 17:48:15.763811 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:15.763813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:15.763820 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:48:15.763856 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:15.763857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:15.763879 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:15.763944 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:48:15.764142 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.764830 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.765890 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:15.766218 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:15.766279 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:15.766494 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:48:15.766619 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:48:15.766696 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:15.766922 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 17:48:15.766943 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 17:48:15.772310 15594 raft_consensus.cc:1261] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [term 1 FOLLOWER]: Refusing update from remote peer 151fc31043af47da9b8fb75ebd5413dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:48:15.772692 15666 consensus_queue.cc:1035] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42cf4debec64ee184a375be54256728" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 45627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:48:15.774786 15324 raft_consensus.cc:1261] T 7914d3d6294a482d88a08846d137ca7d P 54a1fb5eaab447eea773a364a79d0072 [term 1 FOLLOWER]: Refusing update from remote peer 151fc31043af47da9b8fb75ebd5413dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:48:15.775038 15666 consensus_queue.cc:1035] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "54a1fb5eaab447eea773a364a79d0072" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:48:15.778903 15673 mvcc.cc:205] Tried to move back new op lower bound from 7334370081880621056 to 7334370081745080320. Current Snapshot: MvccSnapshot[applied={T|T < 7334370081880621056}]
I20260928 17:48:15.779026 15674 mvcc.cc:205] Tried to move back new op lower bound from 7334370081880621056 to 7334370081745080320. Current Snapshot: MvccSnapshot[applied={T|T < 7334370081880621056}]
I20260928 17:48:15.781113 15676 mvcc.cc:205] Tried to move back new op lower bound from 7334370081880621056 to 7334370081745080320. Current Snapshot: MvccSnapshot[applied={T|T < 7334370081880621056}]
I20260928 17:48:15.784032 15157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:48:15.784168 15157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:48:15.784855 15157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24544f3089644d6fa60b9d12fd6e5d15: Sending DeleteTablet for 3 replicas of tablet 7914d3d6294a482d88a08846d137ca7d
I20260928 17:48:15.785328 15576 tablet_service.cc:1514] Processing DeleteTablet for tablet 7914d3d6294a482d88a08846d137ca7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:48:15 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:47962
I20260928 17:48:15.785377 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 7914d3d6294a482d88a08846d137ca7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:48:15 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:43504
I20260928 17:48:15.785585 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet 7914d3d6294a482d88a08846d137ca7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:48:15 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:42564
I20260928 17:48:15.786104 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 15239
I20260928 17:48:15.794013 15683 tablet_replica.cc:330] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: stopping tablet replica
I20260928 17:48:15.794122 15683 raft_consensus.cc:2229] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:48:15.794245 15683 raft_consensus.cc:2258] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:48:15.794683 15683 ts_tablet_manager.cc:1905] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:48:15.795310 15682 tablet_replica.cc:330] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: stopping tablet replica
I20260928 17:48:15.795480 15682 raft_consensus.cc:2229] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:48:15.795612 15682 raft_consensus.cc:2258] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:48:15.796048 15682 ts_tablet_manager.cc:1905] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:48:15.799582 15683 ts_tablet_manager.cc:1918] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:48:15.799674 15683 log.cc:1196] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/7914d3d6294a482d88a08846d137ca7d
I20260928 17:48:15.799969 15683 ts_tablet_manager.cc:1939] T 7914d3d6294a482d88a08846d137ca7d P 151fc31043af47da9b8fb75ebd5413dd: Deleting consensus metadata
I20260928 17:48:15.800405 15141 catalog_manager.cc:4841] TS 151fc31043af47da9b8fb75ebd5413dd (127.3.222.2:36219): tablet 7914d3d6294a482d88a08846d137ca7d (table test-table [id=def11ad3aad5471481eb40e86ae8e043]) successfully deleted
I20260928 17:48:15.803784 15682 ts_tablet_manager.cc:1918] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:48:15.803864 15682 log.cc:1196] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/7914d3d6294a482d88a08846d137ca7d
I20260928 17:48:15.804509 15682 ts_tablet_manager.cc:1939] T 7914d3d6294a482d88a08846d137ca7d P f42cf4debec64ee184a375be54256728: Deleting consensus metadata
I20260928 17:48:15.804943 15141 catalog_manager.cc:4841] TS f42cf4debec64ee184a375be54256728 (127.3.222.3:45627): tablet 7914d3d6294a482d88a08846d137ca7d (table test-table [id=def11ad3aad5471481eb40e86ae8e043]) successfully deleted
W20260928 17:48:15.807376 15143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:44025 (error 108)
I20260928 17:48:15.808183 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:15.808075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44025 (local address 127.0.0.1:42572)
0928 17:48:15.808121 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:15.808151 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:44025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 17:48:15.808292 15143 catalog_manager.cc:4584] TS 54a1fb5eaab447eea773a364a79d0072 (127.3.222.1:44025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7914d3d6294a482d88a08846d137ca7d: Network error: Client connection negotiation failed: client connection to 127.3.222.1:44025: connect: Connection refused (error 111)
I20260928 17:48:15.808466 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 15376
I20260928 17:48:15.814754 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 15513
I20260928 17:48:15.820710 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 15125
I20260928 17:48:15.826470 3960 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 17:48:16.160151 3960 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 17:48:16.160240 3960 mini_ranger.cc:66] Stopping Ranger...
I20260928 17:48:16.498667 3960 mini_ranger.cc:68] Stopped Ranger
2026-09-28 17:48:16.498 UTC [14711] LOG: received smart shutdown request
2026-09-28 17:48:16.502 UTC [14711] LOG: background worker "logical replication launcher" (PID 14720) exited with exit code 1
2026-09-28 17:48:16.502 UTC [14715] LOG: shutting down
2026-09-28 17:48:16.677 UTC [14711] LOG: database system is shut down
2026-09-28T17:48:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43482 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:48:16 dist-test-slave-w3bm krb5kdc[15694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:48:16 dist-test-slave-w3bm krb5kdc[15694](info): set up 2 sockets
Sep 28 17:48:16 dist-test-slave-w3bm krb5kdc[15694](info): commencing operation
krb5kdc: starting...
W20260928 17:48:17.702791 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:48:17 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:48:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:48:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/rangeradmin_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/rangeradmin_127.3.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/rangerlookup_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/rangerlookup_127.3.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/HTTP_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/HTTP_127.3.222.20@KRBTEST.COM.keytab.
I20260928 17:48:17.860253 3960 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkw9tqR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/postgres -l logfile start
2026-09-28 17:48:20.303 UTC [15724] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 17:48:20.303 UTC [15724] LOG: listening on IPv4 address "127.3.222.20", port 35045
2026-09-28 17:48:20.307 UTC [15724] LOG: listening on Unix socket "/tmp/.s.PGSQL.35045"
2026-09-28 17:48:20.334 UTC [15727] LOG: database system was shut down at 2026-09-28 17:48:18 UTC
2026-09-28 17:48:20.338 UTC [15724] LOG: database system is ready to accept connections
I20260928 17:48:21.210897 3960 mini_postgres.cc:96] Postgres bound to 35045
2026-09-28 17:48:21.216 UTC [15735] FATAL: database "slave" does not exist
127.3.222.20:35045 - accepting connections
I20260928 17:48:21.216943 3960 mini_ranger.cc:161] Starting Ranger...
I20260928 17:48:21.230985 3960 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 17:48:21.726221 3960 mini_ranger.cc:87] Created ranger Postgres database
I20260928 17:48:21.726305 3960 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-admin
2026-09-28 17:48:21,996 [I] DB FLAVOR :POSTGRES
2026-09-28 17:48:21,996 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 17:48:21,996 [I] Checking connection..
2026-09-28 17:48:21,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 17:48:22,247 [I] Checking connection passed.
2026-09-28 17:48:22,247 [I] --------- Verifying version history table ---------
2026-09-28 17:48:22,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:48:22,485 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 17:48:22,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:48:22,716 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 17:48:22,716 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 17:48:22,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 17:48:22.963 UTC [15799] WARNING: there is no transaction in progress
2026-09-28 17:48:22,969 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 17:48:22,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:48:23,220 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 17:48:23,221 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 17:48:23,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 17:48:23,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 17:48:23,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w3bm.c.gcp-upstream.internal','N') ;"
2026-09-28 17:48:23,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 17:48:24,176 [I] Table x_portal_user does not exist in database ranger
2026-09-28 17:48:24,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 17:48:24,420 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 17:48:24,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 17:48:24,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 17:48:24,885 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 17:48:24,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 17:48:25.847 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:25.864 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:25.880 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:25.895 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:25.910 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:25.925 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:25.977 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:25.984 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:25.999 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:26.008 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:26.018 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:26.033 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:26.043 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:26.049 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:26.056 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:26.062 UTC [15907] WARNING: there is no transaction in progress
2026-09-28 17:48:26,729 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 17:48:26,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 17:48:26,952 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 17:48:26,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 17:48:27,201 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 17:48:27,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 17:48:27,450 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 17:48:27,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 17:48:27,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 17:48:27,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w3bm.c.gcp-upstream.internal';"
2026-09-28 17:48:28,150 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 17:48:28,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 17:48:28,376 [I] DB_PATCHES have already been applied
I20260928 17:48:28.382089 3960 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkw9tqR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 17:48:28.382148 3960 mini_ranger.cc:193] Using host: 127.3.222.20
I20260928 17:48:28.384960 3960 mini_ranger.cc:227] Ranger admin URL: http://127.3.222.20:48413
Sep 28, 2026 5:48:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:48:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:48:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:48:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:48:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:48:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:48:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 17:48:29.368090 3960 mini_ranger.cc:160] Time spent starting Ranger: real 8.151s user 0.000s sys 0.008s
Sep 28, 2026 5:48:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:48:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 5:48:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:48:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:48:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 17:48:49.532657 3960 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/rangerkms_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/rangerkms_127.3.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.222.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.222.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.222.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/HTTP_127.3.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.222.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/HTTP_127.3.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 17:48:49 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617729, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 17:48:49.614789 3960 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 17:48:49.623147 3960 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 17:48:50.525120 3960 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 17:48:50.525199 3960 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-kms
I20260928 17:48:50.525215 3960 mini_ranger_kms.cc:227] Using postgres at 127.3.222.20:35045
2026-09-28 17:48:50,667 [I] DB FLAVOR :POSTGRES
2026-09-28 17:48:50,667 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 17:48:50,668 [I] Checking connection
2026-09-28 17:48:50,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 17:48:50,888 [I] connection success
2026-09-28 17:48:50,888 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 17:48:50,888 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 17:48:50,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 17:48:51,134 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 17:48:51,134 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 17:48:51,135 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 17:48:51,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.222.20:35045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 17:48:51,380 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 17:48:51.818311 3960 mini_ranger_kms.cc:310] Created kms service
I20260928 17:48:52.003355 3960 mini_ranger_kms.cc:326] Created kudu user
I20260928 17:48:52.046540 3960 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 17:48:52 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617729, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:52.398046 3960 mini_ranger_kms.cc:384] Added ranger policy
I20260928 17:48:52.398277 3960 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkw9tqR/build/release/bin/postgresql.jar:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkw9tqR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 17:48:52.398303 3960 mini_ranger_kms.cc:240] Using host: 127.3.222.20
I20260928 17:48:52.402132 3960 mini_ranger_kms.cc:276] Ranger KMS PID: 16087
I20260928 17:48:52.402194 3960 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.222.20:34259
Sep 28, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 17:48:54.430203 3960 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.815s user 0.000s sys 0.126s
I20260928 17:48:54.430320 3960 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 17:48:54.430368 3960 mini_ranger_kms.cc:401] 127.3.222.20:34259/kms/v1/keys
Sep 28 17:48:54 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790617734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.222.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 17:48:54 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.222.20@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
Sep 28 17:48:55 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.222.20@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
Sep 28 17:48:57 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:48:57.632138 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35061
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:35061
--ranger_config_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:34259/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:48:57.709197 16138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:57.709378 16138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:57.709398 16138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:57.710475 16138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:48:57.710513 16138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:57.710526 16138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:57.710538 16138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:48:57.710561 16138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:48:57.712069 16138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.222.20:34259/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:35061
--ranger_config_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:57.712275 16138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:57.712463 16138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:57.717957 16144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:57.718052 16138 server_base.cc:1034] running on GCE node
W20260928 17:48:57.717998 16146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:57.718146 16143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:57.718434 16138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:57.718693 16138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:57.719828 16138 hybrid_clock.cc:648] HybridClock initialized: now 1790617737719815 us; error 28 us; skew 500 ppm
Sep 28 17:48:57 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:57.727849 16138 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:48:57.728286 16138 webserver.cc:460] Webserver started at http://127.3.222.62:44655/ using document root <none> and password file <none>
I20260928 17:48:57.728502 16138 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:57.728539 16138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:48:57.728647 16138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:48:57 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:57.789242 16138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "ca346f5006e041b5a981c56367852cae"
format_stamp: "Formatted at 2026-09-28 17:48:57 on dist-test-slave-w3bm"
server_key: "0ab3015b248326f6d95effaf5ece41fd"
server_key_iv: "912809815ef60bb0164931be2e5d21ae"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:57.789638 16138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "ca346f5006e041b5a981c56367852cae"
format_stamp: "Formatted at 2026-09-28 17:48:57 on dist-test-slave-w3bm"
server_key: "0ab3015b248326f6d95effaf5ece41fd"
server_key_iv: "912809815ef60bb0164931be2e5d21ae"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:57.790943 16138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:48:57.806054 16153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:57.806272 16138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:48:57.806375 16138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "ca346f5006e041b5a981c56367852cae"
format_stamp: "Formatted at 2026-09-28 17:48:57 on dist-test-slave-w3bm"
server_key: "0ab3015b248326f6d95effaf5ece41fd"
server_key_iv: "912809815ef60bb0164931be2e5d21ae"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:57.806473 16138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:57.823006 16138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:57.824927 16138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:57.825060 16138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:57.829665 16138 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:35061
I20260928 17:48:57.829694 16205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:35061 every 8 connection(s)
I20260928 17:48:57.830084 16138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:48:57.830735 16206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:48:57.832957 16206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Bootstrap starting.
I20260928 17:48:57.833478 16206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Neither blocks nor log segments found. Creating new log.
I20260928 17:48:57.833729 16206 log.cc:824] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Log is configured to *not* fsync() on all Append() calls
I20260928 17:48:57.834306 16206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: No bootstrap required, opened a new log
I20260928 17:48:57.836231 16206 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } }
I20260928 17:48:57.836369 16206 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:48:57.836400 16206 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca346f5006e041b5a981c56367852cae, State: Initialized, Role: FOLLOWER
I20260928 17:48:57.836532 16206 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } }
I20260928 17:48:57.836603 16206 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:48:57.836642 16206 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:48:57.836685 16206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:48:57.836984 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 16138
I20260928 17:48:57.837085 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:48:57.837226 16206 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } }
I20260928 17:48:57.837253 3960 external_mini_cluster.cc:1428] Setting key 20992b710ea90cdcf374d58574e46bd7
I20260928 17:48:57.837299 16206 leader_election.cc:302] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca346f5006e041b5a981c56367852cae; no voters:
I20260928 17:48:57.837463 16209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:48:57.837615 16209 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 1 LEADER]: Becoming Leader. State: Replica: ca346f5006e041b5a981c56367852cae, State: Running, Role: LEADER
I20260928 17:48:57.837760 16209 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } }
I20260928 17:48:57.837757 16206 leader_election.cc:288] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:48:57.838135 16210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca346f5006e041b5a981c56367852cae. Latest consensus state: current_term: 1 leader_uuid: "ca346f5006e041b5a981c56367852cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } } }
I20260928 17:48:57.838225 16209 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca346f5006e041b5a981c56367852cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } } }
I20260928 17:48:57.838452 16209 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: This master's current role is: LEADER
I20260928 17:48:57.838389 16206 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:48:57.839052 16210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: This master's current role is: LEADER
I20260928 17:48:57.839259 16214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:48:57.839797 16214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:48:57.840032 16206 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 17:48:57.843966 16214 catalog_manager.cc:1347] Generated new cluster ID: 05417b60dca84af09b7caee25ec14b0d
I20260928 17:48:57.844039 16214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:48:57.850554 16214 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:48:57.851382 16214 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 17:48:57 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:48:57.865218 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:57.837677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.3.222.62:35061)
0928 17:48:57.837848 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:57.837858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:57.837895 (+ 37us) server_negotiation.cc:407] Connection header received
0928 17:48:57.838041 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:57.838047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:57.838132 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:57.838257 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:57.839088 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:57.849435 (+ 10347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:57.857493 (+ 8058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:57.858223 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:57.858675 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:57.861212 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:57.861218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:57.861218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:57.861232 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:57.863120 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:57.863557 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:57.863561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:57.863561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:57.863662 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:57.863861 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:57.863862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:57.863862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:57.864158 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:57.864737 (+ 579us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:57.864815 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 17:48:57.864972 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260928 17:48:57.866321 16214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Generated new TSK 0
I20260928 17:48:57.866554 16214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:48:58.635818 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:34259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35061
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:48:58.751024 16250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:58.751410 16250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:58.751488 16250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:58.753190 16250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:58.753286 16250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:58.753381 16250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:48:58.755591 16250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.222.20:34259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:58.755825 16250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:58.756088 16250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:58.762537 16255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:58.762871 16258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:58.763127 16256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:58.764258 16250 server_base.cc:1034] running on GCE node
I20260928 17:48:58.764460 16250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:58.764707 16250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:58.765856 16250 hybrid_clock.cc:648] HybridClock initialized: now 1790617738765834 us; error 35 us; skew 500 ppm
Sep 28 17:48:58 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:58.769275 16250 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:48:58.769781 16250 webserver.cc:460] Webserver started at http://127.3.222.1:40803/ using document root <none> and password file <none>
I20260928 17:48:58.770066 16250 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:58.770133 16250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:48:58.770275 16250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:48:58 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:58.784067 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "83a55beccbc24c3598b7e68d5371c3da"
format_stamp: "Formatted at 2026-09-28 17:48:58 on dist-test-slave-w3bm"
server_key: "d91d65b29edf2f06c924ead2aaa127ef"
server_key_iv: "8db080d89f4bf0c2f7247d82137306c8"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:58.784500 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "83a55beccbc24c3598b7e68d5371c3da"
format_stamp: "Formatted at 2026-09-28 17:48:58 on dist-test-slave-w3bm"
server_key: "d91d65b29edf2f06c924ead2aaa127ef"
server_key_iv: "8db080d89f4bf0c2f7247d82137306c8"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:58.786298 16250 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 17:48:58.807632 16267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:58.807880 16250 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.001s
I20260928 17:48:58.807969 16250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "83a55beccbc24c3598b7e68d5371c3da"
format_stamp: "Formatted at 2026-09-28 17:48:58 on dist-test-slave-w3bm"
server_key: "d91d65b29edf2f06c924ead2aaa127ef"
server_key_iv: "8db080d89f4bf0c2f7247d82137306c8"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:58.808055 16250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:58.822762 16250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:58.824584 16250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:58.824712 16250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:58.824925 16250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:48:58.825255 16250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:48:58.825290 16250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:58.825318 16250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:48:58.825357 16250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:58.831059 16250 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:39677
I20260928 17:48:58.831565 16250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:48:58.833886 16380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:39677 every 8 connection(s)
I20260928 17:48:58.835431 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 16250
I20260928 17:48:58.835521 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:48:58.835686 3960 external_mini_cluster.cc:1428] Setting key f3374f98b4f5052ce30ec0f8808b0dc5
Sep 28 17:48:58 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:48:58.845185 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:58.834478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35061 (local address 127.3.222.1:38927)
0928 17:48:58.835103 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:58.835149 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:58.835967 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:58.836286 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:58.836291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:58.836520 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:48:58.836790 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:58.836810 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:58.837705 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:58.837707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:58.838967 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:58.838970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:58.839522 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:58.839524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:58.839698 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:58.840210 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:48:58.840220 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:48:58.841771 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:48:58.843659 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:58.843665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:58.843668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:58.843945 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:58.844147 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:58.844148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:58.844148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:58.844267 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:58.844536 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:48:58.844541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:48:58.844708 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 17:48:58.844818 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 17:48:58.844962 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":86,"threads_started":1}
I20260928 17:48:58.845709 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:58.834780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:38927 (local address 127.3.222.62:35061)
0928 17:48:58.835402 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:58.835405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:58.835416 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:48:58.836108 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:58.836109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:58.836120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:58.836172 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:58.836905 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:58.837575 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:58.839056 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:58.839387 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:58.839445 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:58.841882 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:58.841883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:58.841883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:58.841890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:58.843574 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:58.844023 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:58.844024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:58.844025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:58.844086 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:58.844330 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:58.844331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:58.844331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:58.844461 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:58.844514 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:58.845534 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0928 17:48:58.845615 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":81,"threads_started":1}
I20260928 17:48:58.846393 16381 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35061
I20260928 17:48:58.846478 16381 heartbeater.cc:464] Registering TS with master...
I20260928 17:48:58.846644 16381 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
I20260928 17:48:58.847193 16170 ts_manager.cc:194] Registered new tserver with Master: 83a55beccbc24c3598b7e68d5371c3da (127.3.222.1:39677)
I20260928 17:48:58.848083 16170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:38927
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:48:58.894662 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:34259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35061
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 17:48:59.014415 16388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:59.014741 16388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:59.014812 16388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:59.016580 16388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:59.016696 16388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:59.016881 16388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:48:59.019040 16388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.222.20:34259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:59.019515 16388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:59.019821 16388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:59.030143 16395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:59.030489 16388 server_base.cc:1034] running on GCE node
W20260928 17:48:59.030750 16394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:59.032253 16397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:59.036371 16388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:59.036691 16388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:59.037874 16388 hybrid_clock.cc:648] HybridClock initialized: now 1790617739037856 us; error 36 us; skew 500 ppm
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:59.041013 16388 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:48:59.041520 16388 webserver.cc:460] Webserver started at http://127.3.222.2:43823/ using document root <none> and password file <none>
I20260928 17:48:59.041770 16388 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:59.041826 16388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:48:59.041950 16388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:59.063613 16388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "9908bc72150a4c2bb48eee3c99b9e5d7"
format_stamp: "Formatted at 2026-09-28 17:48:59 on dist-test-slave-w3bm"
server_key: "c72a29fc3eb1f92b66a963801987bcd2"
server_key_iv: "79ddf8b5ad9664b31725004c63efea0b"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:59.064013 16388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "9908bc72150a4c2bb48eee3c99b9e5d7"
format_stamp: "Formatted at 2026-09-28 17:48:59 on dist-test-slave-w3bm"
server_key: "c72a29fc3eb1f92b66a963801987bcd2"
server_key_iv: "79ddf8b5ad9664b31725004c63efea0b"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:59.065972 16388 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 17:48:59.079629 16404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.079825 16388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:48:59.079928 16388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "9908bc72150a4c2bb48eee3c99b9e5d7"
format_stamp: "Formatted at 2026-09-28 17:48:59 on dist-test-slave-w3bm"
server_key: "c72a29fc3eb1f92b66a963801987bcd2"
server_key_iv: "79ddf8b5ad9664b31725004c63efea0b"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:59.080039 16388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:59.100245 16388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:59.102592 16388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:59.102787 16388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:59.103075 16388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:48:59.103509 16388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:48:59.103596 16388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.103663 16388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:48:59.103716 16388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.109786 16388 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:44253
I20260928 17:48:59.110280 16388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:48:59.113287 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 16388
I20260928 17:48:59.113408 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:48:59.112954 16518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:44253 every 8 connection(s)
I20260928 17:48:59.113602 3960 external_mini_cluster.cc:1428] Setting key ed0003d6149bd3014c8349aa33ad96f8
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:48:59.135520 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:59.113870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:49283 (local address 127.3.222.62:35061)
0928 17:48:59.113915 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:59.113917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:59.113926 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:48:59.115315 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:59.115317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:59.115329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:59.115386 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:59.124974 (+ 9588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.125777 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.129472 (+ 3695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.129810 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.129889 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:59.132108 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:59.132110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:59.132110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:59.132118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:59.133837 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:59.134313 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:59.134314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:59.134315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:59.134386 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:59.134693 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:59.134695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:59.134695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:59.134830 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:59.134880 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:59.135302 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 17:48:59.135400 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:48:59.136258 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:59.113585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35061 (local address 127.3.222.2:49283)
0928 17:48:59.113754 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:59.113803 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:59.114427 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:59.124293 (+ 9866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:59.124300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:59.124551 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:48:59.124852 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:59.124859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.126366 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.126367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:59.129384 (+ 3017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:59.129388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.129933 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.129934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:59.130084 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:59.130608 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:48:59.130619 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:48:59.132003 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:48:59.133930 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:59.133938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:59.133940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:59.134222 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:59.134454 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:59.134455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:59.134455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:59.134613 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:59.134911 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:48:59.134917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:48:59.135106 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 17:48:59.135862 (+ 756us) client_negotiation.cc:240] Negotiation successful
0928 17:48:59.136023 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260928 17:48:59.138329 16519 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35061
I20260928 17:48:59.138443 16519 heartbeater.cc:464] Registering TS with master...
I20260928 17:48:59.138628 16519 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
I20260928 17:48:59.139041 16170 ts_manager.cc:194] Registered new tserver with Master: 9908bc72150a4c2bb48eee3c99b9e5d7 (127.3.222.2:44253)
I20260928 17:48:59.139698 16170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:49283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:48:59.154932 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:34259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35061
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:48:59.274719 16524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:59.275070 16524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:59.275156 16524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:59.276859 16524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:59.276952 16524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:59.277011 16524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:48:59.278942 16524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.222.20:34259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:59.279175 16524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:59.279417 16524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:59.285128 16529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:59.285140 16530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:59.285240 16524 server_base.cc:1034] running on GCE node
W20260928 17:48:59.285157 16532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:59.285607 16524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:59.285836 16524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:59.286978 16524 hybrid_clock.cc:648] HybridClock initialized: now 1790617739286966 us; error 27 us; skew 500 ppm
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:59.289359 16524 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:48:59.289781 16524 webserver.cc:460] Webserver started at http://127.3.222.3:34331/ using document root <none> and password file <none>
I20260928 17:48:59.289983 16524 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:59.290030 16524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:48:59.290146 16524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:59.302348 16524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "e34f283b310d4fd689877e9f061bde7b"
format_stamp: "Formatted at 2026-09-28 17:48:59 on dist-test-slave-w3bm"
server_key: "d3eb2fb03d1c6bd3a56ea79773523793"
server_key_iv: "1fa6987cab4aafd1abb73458c908c19f"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:59.302695 16524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "e34f283b310d4fd689877e9f061bde7b"
format_stamp: "Formatted at 2026-09-28 17:48:59 on dist-test-slave-w3bm"
server_key: "d3eb2fb03d1c6bd3a56ea79773523793"
server_key_iv: "1fa6987cab4aafd1abb73458c908c19f"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:59.316269 16524 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.002s sys 0.000s
I20260928 17:48:59.331746 16539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.332055 16524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:48:59.332154 16524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "e34f283b310d4fd689877e9f061bde7b"
format_stamp: "Formatted at 2026-09-28 17:48:59 on dist-test-slave-w3bm"
server_key: "d3eb2fb03d1c6bd3a56ea79773523793"
server_key_iv: "1fa6987cab4aafd1abb73458c908c19f"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:59.332247 16524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:59.363341 16524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:59.365676 16524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:59.365885 16524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:59.366143 16524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:48:59.366510 16524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:48:59.366562 16524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.366592 16524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:48:59.366665 16524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.373243 16524 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:35859
I20260928 17:48:59.373641 16524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:48:59.375558 16652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:35859 every 8 connection(s)
I20260928 17:48:59.382730 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 16524
I20260928 17:48:59.382819 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:48:59.382995 3960 external_mini_cluster.cc:1428] Setting key f9c1059a173641f98f448dbd59781db9
I20260928 17:48:59.386541 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:59.376952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:34875 (local address 127.3.222.62:35061)
0928 17:48:59.377006 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:59.377008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:59.377018 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:48:59.377601 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:59.377602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:59.377614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:59.377676 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:59.378353 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.379139 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.380599 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.380899 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.380975 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:59.383452 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:59.383454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:59.383454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:59.383462 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:59.385035 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:59.385470 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:59.385472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:59.385472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:59.385533 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:59.385826 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:59.385827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:59.385828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:59.385955 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:59.385995 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:59.386366 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 17:48:59.386446 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:48:59.386706 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:59.376651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35061 (local address 127.3.222.3:34875)
0928 17:48:59.376827 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:59.376879 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:59.377479 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:59.377748 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:59.377752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:59.377987 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:48:59.378251 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:59.378257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.379281 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.379282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:59.380502 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:59.380505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.380997 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.380998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:59.381142 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:59.381713 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:48:59.381722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:48:59.383344 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:48:59.385107 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:59.385113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:59.385115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:59.385396 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:59.385610 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:59.385611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:59.385612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:59.385751 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:59.386039 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:48:59.386045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:48:59.386210 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 17:48:59.386336 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 17:48:59.386480 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260928 17:48:59.387194 16653 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35061
I20260928 17:48:59.387315 16653 heartbeater.cc:464] Registering TS with master...
I20260928 17:48:59.387492 16653 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
I20260928 17:48:59.387850 16170 ts_manager.cc:194] Registered new tserver with Master: e34f283b310d4fd689877e9f061bde7b (127.3.222.3:35859)
I20260928 17:48:59.388334 16170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:34875
I20260928 17:48:59.394582 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:48:59.395197 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 16250
I20260928 17:48:59.401873 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 16388
I20260928 17:48:59.406812 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 16524
I20260928 17:48:59.412091 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 16138
I20260928 17:48:59.416204 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35061
--webserver_interface=127.3.222.62
--webserver_port=44655
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:35061
--ranger_config_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:34259/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.222.20:34259/kms with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:48:59.493570 16656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:59.493758 16656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:59.493778 16656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:59.494867 16656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:48:59.494904 16656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:59.494917 16656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:59.494927 16656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:48:59.494937 16656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:48:59.496492 16656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.222.20:34259/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:35061
--ranger_config_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=44655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:59.496698 16656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:59.496904 16656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:59.501356 16662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:59.501370 16664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:59.501467 16661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:59.501610 16656 server_base.cc:1034] running on GCE node
I20260928 17:48:59.501827 16656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:59.502059 16656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:59.503201 16656 hybrid_clock.cc:648] HybridClock initialized: now 1790617739503185 us; error 30 us; skew 500 ppm
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:59.505468 16656 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:48:59.505879 16656 webserver.cc:460] Webserver started at http://127.3.222.62:44655/ using document root <none> and password file <none>
I20260928 17:48:59.506072 16656 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:59.506120 16656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:59.554188 16656 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:48:59.554993 16672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.555158 16656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:48:59.555287 16656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "ca346f5006e041b5a981c56367852cae"
format_stamp: "Formatted at 2026-09-28 17:48:57 on dist-test-slave-w3bm"
server_key: "0ab3015b248326f6d95effaf5ece41fd"
server_key_iv: "912809815ef60bb0164931be2e5d21ae"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:59.555742 16656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:59.585608 16656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:59.587625 16656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:59.587782 16656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:59.592460 16656 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:35061
I20260928 17:48:59.592720 16724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:35061 every 8 connection(s)
I20260928 17:48:59.592914 16656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:48:59.594404 16725 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:48:59.595274 16725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Bootstrap starting.
I20260928 17:48:59.601768 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 16656
I20260928 17:48:59.602047 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:34259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:39677
--local_ip_for_outbound_sockets=127.3.222.1
--tserver_master_addrs=127.3.222.62:35061
--webserver_port=40803
--webserver_interface=127.3.222.1
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.222.20:34259/kms with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:48:59.605206 16725 log.cc:824] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Log is configured to *not* fsync() on all Append() calls
I20260928 17:48:59.607455 16725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:48:59.607753 16725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Bootstrap complete.
I20260928 17:48:59.610999 16725 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } }
I20260928 17:48:59.611351 16725 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca346f5006e041b5a981c56367852cae, State: Initialized, Role: FOLLOWER
I20260928 17:48:59.611538 16725 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } }
I20260928 17:48:59.611608 16725 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:48:59.611670 16725 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:48:59.611716 16725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:48:59.613027 16725 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } }
I20260928 17:48:59.613132 16725 leader_election.cc:302] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca346f5006e041b5a981c56367852cae; no voters:
I20260928 17:48:59.613305 16725 leader_election.cc:288] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 17:48:59.613363 16729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:48:59.613575 16729 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [term 2 LEADER]: Becoming Leader. State: Replica: ca346f5006e041b5a981c56367852cae, State: Running, Role: LEADER
I20260928 17:48:59.613659 16725 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:48:59.613709 16729 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } }
I20260928 17:48:59.613960 16729 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca346f5006e041b5a981c56367852cae. Latest consensus state: current_term: 2 leader_uuid: "ca346f5006e041b5a981c56367852cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } } }
I20260928 17:48:59.613958 16731 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca346f5006e041b5a981c56367852cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca346f5006e041b5a981c56367852cae" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35061 } } }
I20260928 17:48:59.614056 16729 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: This master's current role is: LEADER
I20260928 17:48:59.614063 16731 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae [sys.catalog]: This master's current role is: LEADER
I20260928 17:48:59.614218 16733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:48:59.614674 16733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:48:59.614835 16733 catalog_manager.cc:1259] Loaded cluster ID: 05417b60dca84af09b7caee25ec14b0d
I20260928 17:48:59.614905 16733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:48:59.615640 16725 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 17:48:59.616168 16733 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:48:59.616401 16733 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Loaded TSK: 0
I20260928 17:48:59.616705 16733 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 17:48:59.714064 16727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:59.714411 16727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:59.714488 16727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:59.716261 16727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:59.716370 16727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:59.716451 16727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:48:59.718595 16727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.222.20:34259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:39677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=40803
--tserver_master_addrs=127.3.222.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:59.719033 16727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:59.719333 16727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:59.725598 16765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:59.725984 16763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:59.726681 16727 server_base.cc:1034] running on GCE node
W20260928 17:48:59.727346 16762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:59.731680 16727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:59.731953 16727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:59.733109 16727 hybrid_clock.cc:648] HybridClock initialized: now 1790617739733095 us; error 31 us; skew 500 ppm
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:59.737043 16727 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:48:59.737555 16727 webserver.cc:460] Webserver started at http://127.3.222.1:40803/ using document root <none> and password file <none>
I20260928 17:48:59.737792 16727 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:59.737871 16727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:59.765767 16727 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260928 17:48:59.767622 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.767829 16727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 17:48:59.767906 16727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "83a55beccbc24c3598b7e68d5371c3da"
format_stamp: "Formatted at 2026-09-28 17:48:58 on dist-test-slave-w3bm"
server_key: "d91d65b29edf2f06c924ead2aaa127ef"
server_key_iv: "8db080d89f4bf0c2f7247d82137306c8"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:59.768342 16727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:48:59.794955 16727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:48:59.797191 16727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:48:59.797392 16727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:48:59.797677 16727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:48:59.798079 16727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:48:59.798166 16727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.798272 16727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:48:59.798349 16727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.804515 16727 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:39677
I20260928 17:48:59.805079 16727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:48:59.805912 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:39677 every 8 connection(s)
I20260928 17:48:59.809597 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 16727
I20260928 17:48:59.809860 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:34259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:44253
--local_ip_for_outbound_sockets=127.3.222.2
--tserver_master_addrs=127.3.222.62:35061
--webserver_port=43823
--webserver_interface=127.3.222.2
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.222.20:34259/kms with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:48:59.818117 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:59.806684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35061 (local address 127.3.222.1:57621)
0928 17:48:59.807124 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 17:48:59.807182 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 17:48:59.807695 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:48:59.808227 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:48:59.808232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:48:59.808430 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:48:59.808699 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:59.808707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.810074 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.810075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:59.811202 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:48:59.811225 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.812120 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.812122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:48:59.812242 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:59.812760 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:48:59.812769 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:48:59.814227 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:48:59.816304 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:59.816310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:59.816312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:59.816583 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:59.816866 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:48:59.816867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:48:59.816867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:48:59.816994 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:48:59.817465 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:48:59.817472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:48:59.817624 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 17:48:59.817746 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 17:48:59.817890 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":72,"threads_started":1}
I20260928 17:48:59.818606 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:48:59.806755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:57621 (local address 127.3.222.62:35061)
0928 17:48:59.806935 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 17:48:59.806946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:48:59.807854 (+ 908us) server_negotiation.cc:407] Connection header received
0928 17:48:59.808003 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:48:59.808009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:48:59.808053 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:48:59.808182 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:48:59.808833 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.809965 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.811322 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:48:59.811909 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:48:59.812010 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:48:59.814343 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:48:59.814348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:48:59.814348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:48:59.814361 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:48:59.816213 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:59.816683 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:59.816687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:59.816687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:59.816792 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:48:59.817063 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:48:59.817064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:48:59.817065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:48:59.817357 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:48:59.817435 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:48:59.818298 (+ 863us) server_negotiation.cc:299] Negotiation successful
0928 17:48:59.818418 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260928 17:48:59.819334 16887 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35061
I20260928 17:48:59.819422 16887 heartbeater.cc:464] Registering TS with master...
I20260928 17:48:59.819600 16887 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
W20260928 17:48:59.820437 16887 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:35061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 17:48:59.820500 16887 heartbeater.cc:464] Registering TS with master...
I20260928 17:48:59.820556 16887 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
I20260928 17:48:59.821058 16887 heartbeater.cc:464] Registering TS with master...
I20260928 17:48:59.821130 16887 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
W20260928 17:48:59.821445 16887 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:48:59.936842 16891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:48:59.937213 16891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:48:59.937317 16891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:48:59.939085 16891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:48:59.939332 16891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:48:59.939436 16891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:48:59.941681 16891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.222.20:34259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:44253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=43823
--tserver_master_addrs=127.3.222.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:48:59.942127 16891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:48:59.942469 16891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:48:59.949024 16897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:59.950258 16891 server_base.cc:1034] running on GCE node
W20260928 17:48:59.952308 16896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:48:59.952565 16899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:48:59.952834 16891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:48:59.953114 16891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:48:59.968518 16891 hybrid_clock.cc:648] HybridClock initialized: now 1790617739968486 us; error 55 us; skew 500 ppm
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:48:59.971769 16891 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:48:59.972360 16891 webserver.cc:460] Webserver started at http://127.3.222.2:43823/ using document root <none> and password file <none>
I20260928 17:48:59.972648 16891 fs_manager.cc:359] Metadata directory not provided
I20260928 17:48:59.972733 16891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 17:48:59 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:48:59.990387 16891 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 17:48:59.991314 16907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:48:59.991487 16891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:48:59.991575 16891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "9908bc72150a4c2bb48eee3c99b9e5d7"
format_stamp: "Formatted at 2026-09-28 17:48:59 on dist-test-slave-w3bm"
server_key: "c72a29fc3eb1f92b66a963801987bcd2"
server_key_iv: "79ddf8b5ad9664b31725004c63efea0b"
server_key_version: "kuduclusterkey@0"
I20260928 17:48:59.991978 16891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:00.018231 16891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:00.020645 16891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:00.020823 16891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:00.021080 16891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:00.021466 16891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:00.021516 16891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:00.021546 16891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:00.021608 16891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:00.028724 16891 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:44253
I20260928 17:49:00.028820 17020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:44253 every 8 connection(s)
I20260928 17:49:00.029223 16891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:49:00 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:00.036725 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 16891
I20260928 17:49:00.037012 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.222.20:34259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:35859
--local_ip_for_outbound_sockets=127.3.222.3
--tserver_master_addrs=127.3.222.62:35061
--webserver_port=34331
--webserver_interface=127.3.222.3
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.222.20:34259/kms with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:00.038686 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:00.030069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:37477 (local address 127.3.222.62:35061)
0928 17:49:00.030121 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:00.030123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:00.030317 (+ 194us) server_negotiation.cc:407] Connection header received
0928 17:49:00.030866 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:00.030867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:00.030883 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:00.030934 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:00.031514 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.032003 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.033197 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.033408 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.033461 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:00.035376 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:00.035378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:00.035378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:00.035386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:00.036953 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:00.037577 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:00.037579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:00.037579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:00.037643 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:00.037948 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:00.037948 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:00.037949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:00.038082 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:00.038145 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:00.038506 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 17:49:00.038594 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:49:00.038841 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:00.030062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35061 (local address 127.3.222.2:37477)
0928 17:49:00.030207 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:00.030247 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:00.030775 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:00.031006 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:00.031010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:00.031172 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:00.031424 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:00.031437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.032079 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.032080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:00.033126 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:00.033129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.033493 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.033494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:00.033617 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:00.034071 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:00.034079 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:00.035294 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:00.037048 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:00.037054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:00.037056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:00.037442 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:00.037714 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:00.037715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:00.037715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:00.037879 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:00.038175 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:00.038183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:00.038357 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 17:49:00.038469 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 17:49:00.038614 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":88,"threads_started":1}
I20260928 17:49:00.039664 17021 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35061
I20260928 17:49:00.039762 17021 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:00.039929 17021 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
W20260928 17:49:00.040696 17021 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:35061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 17:49:00.040761 17021 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:00.040820 17021 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
I20260928 17:49:00.041198 17021 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:00.041268 17021 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
W20260928 17:49:00.041584 17021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:49:00.139992 17024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:00.140349 17024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:00.140425 17024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:00.141849 17024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:00.141893 17024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:00.141929 17024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:49:00.143533 17024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.222.20:34259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:35859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=34331
--tserver_master_addrs=127.3.222.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:00.143750 17024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:00.143957 17024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:00.149526 17033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:00.149573 17030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:00.149520 17031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:00.150038 17024 server_base.cc:1034] running on GCE node
I20260928 17:49:00.150233 17024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:00.150458 17024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:00.151604 17024 hybrid_clock.cc:648] HybridClock initialized: now 1790617740151592 us; error 33 us; skew 500 ppm
Sep 28 17:49:00 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:00.153978 17024 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:49:00.154413 17024 webserver.cc:460] Webserver started at http://127.3.222.3:34331/ using document root <none> and password file <none>
I20260928 17:49:00.154618 17024 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:00.154670 17024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 17:49:00 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:49:00.171821 17024 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.001s
I20260928 17:49:00.172820 17041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:00.172988 17024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:49:00.173067 17024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "e34f283b310d4fd689877e9f061bde7b"
format_stamp: "Formatted at 2026-09-28 17:48:59 on dist-test-slave-w3bm"
server_key: "d3eb2fb03d1c6bd3a56ea79773523793"
server_key_iv: "1fa6987cab4aafd1abb73458c908c19f"
server_key_version: "kuduclusterkey@0"
I20260928 17:49:00.173467 17024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:00.181479 17024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:00.183657 17024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:00.183849 17024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:00.184123 17024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:00.184547 17024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:00.184597 17024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:00.184657 17024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:00.184722 17024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:00.191761 17024 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:35859
I20260928 17:49:00.192225 17024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:49:00.192471 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 17024
I20260928 17:49:00.193838 17154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:35859 every 8 connection(s)
Sep 28 17:49:00 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:00.201354 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:00.192912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.3.222.62:35061)
0928 17:49:00.192965 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:00.192967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:00.194429 (+ 1462us) server_negotiation.cc:407] Connection header received
0928 17:49:00.194587 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:00.194588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:00.194601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:00.194659 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:00.195281 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.195902 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.196926 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.197225 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.197304 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:00.198605 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:00.198607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:00.198607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:00.198614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:00.200188 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:00.200528 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:00.200530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:00.200530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:00.200593 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:00.200795 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:00.200796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:00.200796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:00.200928 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:00.201091 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:00.201173 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 17:49:00.201257 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:49:00.206243 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:00.194412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35061 (local address 127.3.222.3:46555)
0928 17:49:00.194656 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:00.194702 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:00.195357 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:00.195619 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:00.195624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:00.195843 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:00.196119 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:00.196126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.197000 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.197001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:00.198108 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:00.198112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.198629 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.198630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:00.198733 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:00.199250 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:00.199259 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:00.200729 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:00.204159 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:00.204168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:00.204170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:00.204483 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:00.205079 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:00.205080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:00.205080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:00.205228 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:00.205524 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:00.205530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:00.205717 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 17:49:00.205832 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 17:49:00.205995 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260928 17:49:00.206754 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:00.194397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:46555 (local address 127.3.222.62:35061)
0928 17:49:00.194593 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:00.194594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:00.194785 (+ 191us) server_negotiation.cc:407] Connection header received
0928 17:49:00.195459 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:00.195460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:00.195474 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:00.195541 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:00.196214 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.196933 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.198205 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:00.198492 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:00.198864 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:00.201121 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:00.201122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:00.201122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:00.201563 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:00.204055 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:00.204929 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:00.204930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:00.204930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:00.204998 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:00.205300 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:00.205301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:00.205302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:00.205442 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:00.205497 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:00.206556 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0928 17:49:00.206656 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260928 17:49:00.207583 17155 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35061
I20260928 17:49:00.207705 17155 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:00.207913 17155 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
W20260928 17:49:00.208647 17155 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:35061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 17:49:00.208729 17155 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:00.208832 17155 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
I20260928 17:49:00.209409 17155 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:00.209499 17155 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
W20260928 17:49:00.209811 17155 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 17:49:00 dist-test-slave-w3bm krb5kdc[15694](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790617740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:00.821617 16887 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:00.821797 16887 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
I20260928 17:49:00.822221 16689 ts_manager.cc:194] Registered new tserver with Master: 83a55beccbc24c3598b7e68d5371c3da (127.3.222.1:39677)
I20260928 17:49:00.823287 16689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:57621
I20260928 17:49:01.041746 17021 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:01.041885 17021 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
I20260928 17:49:01.042126 16689 ts_manager.cc:194] Registered new tserver with Master: 9908bc72150a4c2bb48eee3c99b9e5d7 (127.3.222.2:44253)
I20260928 17:49:01.042657 16689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:37477
Sep 28 17:49:01 dist-test-slave-w3bm krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for HTTP/127.3.222.20@KRBTEST.COM
I20260928 17:49:01.210000 17155 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:01.210161 17155 heartbeater.cc:511] Master 127.3.222.62:35061 requested a full tablet report, sending...
I20260928 17:49:01.210498 16689 ts_manager.cc:194] Registered new tserver with Master: e34f283b310d4fd689877e9f061bde7b (127.3.222.3:35859)
I20260928 17:49:01.211024 16689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:46555
I20260928 17:49:01.214552 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:49:01.223440 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.215595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.3.222.62:35061)
0928 17:49:01.215895 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:01.215897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:01.216022 (+ 125us) server_negotiation.cc:407] Connection header received
0928 17:49:01.216109 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:01.216110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:01.216121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:01.216159 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:01.216863 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.217649 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.218703 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.219003 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.219043 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.220534 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:01.220535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:01.220536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:01.220543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:01.222145 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:01.222530 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:01.222531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:01.222531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:01.222595 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:01.222784 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:01.222784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:01.222785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:01.222917 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:01.222991 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:01.223202 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 17:49:01.223331 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":67,"threads_started":1}
I20260928 17:49:01.226028 16689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:49:01.228729 16689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:49:01.250643 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.241494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:39677 (local address 127.0.0.1:58524)
0928 17:49:01.242569 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:01.242576 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:01.242609 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:01.242886 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:01.242889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:01.242891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:01.242981 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.242983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.245029 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.245030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.249231 (+ 4201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.249234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.250312 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.250313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.250406 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.250422 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 17:49:01.250488 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 17:49:01.250543 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":69,"threads_started":1}
I20260928 17:49:01.252645 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.241309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:44253 (local address 127.0.0.1:52792)
0928 17:49:01.242464 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:01.242473 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:01.242508 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:01.244258 (+ 1750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:01.244259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:01.244261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:01.244311 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.244312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.247410 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.247411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.248584 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.248587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.252382 (+ 3795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.252383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.252454 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.252459 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:49:01.252498 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 17:49:01.252543 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"thread_start_us":78,"threads_started":1}
I20260928 17:49:01.253158 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.241590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.3.222.1:39677)
0928 17:49:01.242418 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:01.242423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:01.242654 (+ 231us) server_negotiation.cc:407] Connection header received
0928 17:49:01.242690 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:01.242692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:01.242708 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:01.242784 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:01.243060 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.244939 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.249317 (+ 4378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.250141 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.250212 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.250275 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:01.253029 (+ 2754us) server_negotiation.cc:299] Negotiation successful
0928 17:49:01.253082 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":44,"threads_started":1}
I20260928 17:49:01.253592 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.241722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.3.222.2:44253)
0928 17:49:01.244031 (+ 2309us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:01.244037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:01.244047 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:49:01.244080 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:01.244082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:01.244097 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:01.244167 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:01.246256 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.247317 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.250989 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.251984 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.252056 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.253455 (+ 1399us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:01.253480 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 17:49:01.253520 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2277,"thread_start_us":74,"threads_started":1}
I20260928 17:49:01.254014 16810 tablet_service.cc:1467] Processing CreateTablet for tablet 9b73c379a39449999176d408f32f77ed (DEFAULT_TABLE table=test-table [id=4d493ecd7d6342ff8c783b1b883f6f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:01.254386 16810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b73c379a39449999176d408f32f77ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:01.254316 16955 tablet_service.cc:1467] Processing CreateTablet for tablet 9b73c379a39449999176d408f32f77ed (DEFAULT_TABLE table=test-table [id=4d493ecd7d6342ff8c783b1b883f6f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:01.254601 16955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b73c379a39449999176d408f32f77ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:01.255077 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.241090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35859 (local address 127.0.0.1:46344)
0928 17:49:01.242188 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:01.242230 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:01.242293 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:01.251575 (+ 9282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:01.251576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:01.251579 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:01.251623 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.251625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.252858 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.252860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.253915 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.253918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.254868 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.254869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.254934 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.254938 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:49:01.254967 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 17:49:01.255005 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":84,"threads_started":1}
I20260928 17:49:01.255470 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.241190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.3.222.3:35859)
0928 17:49:01.251310 (+ 10120us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:01.251315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:01.251330 (+ 15us) server_negotiation.cc:407] Connection header received
0928 17:49:01.251366 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:01.251368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:01.251386 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:01.251484 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:01.251691 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.252747 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.254018 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.254759 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.255253 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.255326 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:01.255353 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 17:49:01.255395 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10078,"thread_start_us":47,"threads_started":1}
I20260928 17:49:01.258559 17089 tablet_service.cc:1467] Processing CreateTablet for tablet 9b73c379a39449999176d408f32f77ed (DEFAULT_TABLE table=test-table [id=4d493ecd7d6342ff8c783b1b883f6f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:01.258881 17089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b73c379a39449999176d408f32f77ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:01.259382 17189 tablet_bootstrap.cc:492] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da: Bootstrap starting.
I20260928 17:49:01.260159 17189 tablet_bootstrap.cc:654] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:01.260463 17189 log.cc:824] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:01.260839 17188 tablet_bootstrap.cc:492] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7: Bootstrap starting.
I20260928 17:49:01.261773 17188 tablet_bootstrap.cc:654] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:01.262080 17188 log.cc:824] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:01.263377 17191 tablet_bootstrap.cc:492] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: Bootstrap starting.
I20260928 17:49:01.263965 17191 tablet_bootstrap.cc:654] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:01.264253 17191 log.cc:824] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:01.265064 17189 tablet_bootstrap.cc:492] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da: No bootstrap required, opened a new log
I20260928 17:49:01.265177 17189 ts_tablet_manager.cc:1397] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 17:49:01.265245 17188 tablet_bootstrap.cc:492] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7: No bootstrap required, opened a new log
I20260928 17:49:01.265319 17188 ts_tablet_manager.cc:1397] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 17:49:01.267978 17189 raft_consensus.cc:348] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } }
I20260928 17:49:01.268249 17189 raft_consensus.cc:374] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:01.268334 17189 raft_consensus.cc:729] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a55beccbc24c3598b7e68d5371c3da, State: Initialized, Role: FOLLOWER
I20260928 17:49:01.268519 17189 consensus_queue.cc:260] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } }
I20260928 17:49:01.268858 17189 ts_tablet_manager.cc:1428] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 17:49:01.268800 17188 raft_consensus.cc:348] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } }
I20260928 17:49:01.268961 17188 raft_consensus.cc:374] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:01.269004 17188 raft_consensus.cc:729] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9908bc72150a4c2bb48eee3c99b9e5d7, State: Initialized, Role: FOLLOWER
I20260928 17:49:01.269109 16887 heartbeater.cc:503] Master 127.3.222.62:35061 was elected leader, sending a full tablet report...
I20260928 17:49:01.269136 17188 consensus_queue.cc:260] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } }
I20260928 17:49:01.269385 17188 ts_tablet_manager.cc:1428] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 17:49:01.269493 17021 heartbeater.cc:503] Master 127.3.222.62:35061 was elected leader, sending a full tablet report...
I20260928 17:49:01.270128 17191 tablet_bootstrap.cc:492] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: No bootstrap required, opened a new log
I20260928 17:49:01.270197 17191 ts_tablet_manager.cc:1397] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 17:49:01.272753 17191 raft_consensus.cc:348] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } }
I20260928 17:49:01.272927 17191 raft_consensus.cc:374] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:01.272977 17191 raft_consensus.cc:729] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34f283b310d4fd689877e9f061bde7b, State: Initialized, Role: FOLLOWER
I20260928 17:49:01.273093 17191 consensus_queue.cc:260] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } }
I20260928 17:49:01.273336 17191 ts_tablet_manager.cc:1428] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 17:49:01.273468 17155 heartbeater.cc:503] Master 127.3.222.62:35061 was elected leader, sending a full tablet report...
W20260928 17:49:01.280326 17022 tablet.cc:2369] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:49:01.307132 16888 tablet.cc:2369] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:49:01.313874 17194 raft_consensus.cc:482] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:49:01.314015 17194 raft_consensus.cc:504] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } }
I20260928 17:49:01.314450 17194 leader_election.cc:288] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e34f283b310d4fd689877e9f061bde7b (127.3.222.3:35859), 9908bc72150a4c2bb48eee3c99b9e5d7 (127.3.222.2:44253)
I20260928 17:49:01.318862 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.315542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:57043 (local address 127.3.222.2:44253)
0928 17:49:01.315584 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:01.315586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:01.315592 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:49:01.315614 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:01.315614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:01.315623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:01.315666 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:01.315863 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.316547 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.317964 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.318494 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.318535 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.318546 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:01.318754 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 17:49:01.318795 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:49:01.319043 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.315297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:37225 (local address 127.3.222.3:35859)
0928 17:49:01.315346 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:01.315349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:01.315358 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:49:01.315384 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:01.315385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:01.315397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:01.315448 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:01.315818 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.316542 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.317977 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.318634 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.318686 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.318697 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:01.318932 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 17:49:01.318975 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:49:01.319290 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.314921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:44253 (local address 127.3.222.1:57043)
0928 17:49:01.315367 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:01.315375 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:01.315409 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:01.315713 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:01.315715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:01.315717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:01.315795 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.315797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.316636 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.316637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.317869 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.317872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.318598 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.318599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.318677 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.318681 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:49:01.319133 (+ 452us) client_negotiation.cc:240] Negotiation successful
0928 17:49:01.319175 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":38,"thread_start_us":58,"threads_started":1}
I20260928 17:49:01.319459 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.314727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35859 (local address 127.3.222.1:37225)
0928 17:49:01.315173 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:01.315184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:01.315264 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:01.315639 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:01.315640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:01.315643 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:01.315726 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.315728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.316640 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.316641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.317873 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:01.317876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.318757 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.318758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:01.318842 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.318846 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:49:01.319328 (+ 482us) client_negotiation.cc:240] Negotiation successful
0928 17:49:01.319373 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":17,"thread_start_us":92,"threads_started":1}
I20260928 17:49:01.319710 16968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b73c379a39449999176d408f32f77ed" candidate_uuid: "83a55beccbc24c3598b7e68d5371c3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" is_pre_election: true
I20260928 17:49:01.319873 16968 raft_consensus.cc:2454] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a55beccbc24c3598b7e68d5371c3da in term 0.
I20260928 17:49:01.320087 16777 leader_election.cc:302] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a55beccbc24c3598b7e68d5371c3da, 9908bc72150a4c2bb48eee3c99b9e5d7; no voters:
I20260928 17:49:01.320407 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b73c379a39449999176d408f32f77ed" candidate_uuid: "83a55beccbc24c3598b7e68d5371c3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34f283b310d4fd689877e9f061bde7b" is_pre_election: true
I20260928 17:49:01.320536 17109 raft_consensus.cc:2454] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a55beccbc24c3598b7e68d5371c3da in term 0.
I20260928 17:49:01.320837 17194 raft_consensus.cc:2781] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:49:01.320895 17194 raft_consensus.cc:482] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:49:01.320917 17194 raft_consensus.cc:3037] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:01.321736 17194 raft_consensus.cc:504] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } }
I20260928 17:49:01.321853 17194 leader_election.cc:288] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [CANDIDATE]: Term 1 election: Requested vote from peers e34f283b310d4fd689877e9f061bde7b (127.3.222.3:35859), 9908bc72150a4c2bb48eee3c99b9e5d7 (127.3.222.2:44253)
I20260928 17:49:01.322079 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b73c379a39449999176d408f32f77ed" candidate_uuid: "83a55beccbc24c3598b7e68d5371c3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34f283b310d4fd689877e9f061bde7b"
I20260928 17:49:01.322157 17109 raft_consensus.cc:3037] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:01.322934 17109 raft_consensus.cc:2454] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a55beccbc24c3598b7e68d5371c3da in term 1.
I20260928 17:49:01.323050 16968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b73c379a39449999176d408f32f77ed" candidate_uuid: "83a55beccbc24c3598b7e68d5371c3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7"
I20260928 17:49:01.323125 16968 raft_consensus.cc:3037] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:01.323141 16777 leader_election.cc:302] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a55beccbc24c3598b7e68d5371c3da, e34f283b310d4fd689877e9f061bde7b; no voters:
I20260928 17:49:01.323238 17194 raft_consensus.cc:2781] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:49:01.323365 17194 raft_consensus.cc:686] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [term 1 LEADER]: Becoming Leader. State: Replica: 83a55beccbc24c3598b7e68d5371c3da, State: Running, Role: LEADER
I20260928 17:49:01.323452 17194 consensus_queue.cc:237] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } }
I20260928 17:49:01.323900 16968 raft_consensus.cc:2454] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a55beccbc24c3598b7e68d5371c3da in term 1.
I20260928 17:49:01.324332 16687 catalog_manager.cc:5495] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83a55beccbc24c3598b7e68d5371c3da (127.3.222.1). New cstate: current_term: 1 leader_uuid: "83a55beccbc24c3598b7e68d5371c3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83a55beccbc24c3598b7e68d5371c3da" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 39677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 } health_report { overall_health: UNKNOWN } } }
I20260928 17:49:01.337004 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.333934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.3.222.1:39677)
0928 17:49:01.333979 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:01.333981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:01.333991 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:49:01.334049 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:01.334050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:01.334071 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:01.334142 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:49:01.334279 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.334970 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.336148 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:01.336420 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:01.336486 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:01.336657 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:49:01.336780 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:49:01.336859 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:01.336900 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 17:49:01.336921 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:49:01.338999 16968 raft_consensus.cc:1261] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [term 1 FOLLOWER]: Refusing update from remote peer 83a55beccbc24c3598b7e68d5371c3da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:01.339421 17109 raft_consensus.cc:1261] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [term 1 FOLLOWER]: Refusing update from remote peer 83a55beccbc24c3598b7e68d5371c3da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:01.339756 17199 consensus_queue.cc:1035] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9908bc72150a4c2bb48eee3c99b9e5d7" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 44253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:01.339946 17199 consensus_queue.cc:1035] T 9b73c379a39449999176d408f32f77ed P 83a55beccbc24c3598b7e68d5371c3da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34f283b310d4fd689877e9f061bde7b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:01.344446 17202 mvcc.cc:205] Tried to move back new op lower bound from 7334370268522213376 to 7334370268461596672. Current Snapshot: MvccSnapshot[applied={T|T < 7334370268522213376}]
I20260928 17:49:01.344772 17205 mvcc.cc:205] Tried to move back new op lower bound from 7334370268522213376 to 7334370268461596672. Current Snapshot: MvccSnapshot[applied={T|T < 7334370268522213376}]
I20260928 17:49:01.346709 16687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:01.346791 16687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:01.347556 16687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca346f5006e041b5a981c56367852cae: Sending DeleteTablet for 3 replicas of tablet 9b73c379a39449999176d408f32f77ed
I20260928 17:49:01.347970 17089 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b73c379a39449999176d408f32f77ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:01 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:46344
I20260928 17:49:01.348187 16955 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b73c379a39449999176d408f32f77ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:01 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:52792
I20260928 17:49:01.348531 16810 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b73c379a39449999176d408f32f77ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:01 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:58524
I20260928 17:49:01.349025 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 16727
I20260928 17:49:01.350889 17211 tablet_replica.cc:330] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: stopping tablet replica
I20260928 17:49:01.351014 17212 tablet_replica.cc:330] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7: stopping tablet replica
I20260928 17:49:01.351161 17212 raft_consensus.cc:2229] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:49:01.351327 17211 raft_consensus.cc:2229] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:49:01.351468 17212 raft_consensus.cc:2258] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:01.351591 17211 raft_consensus.cc:2258] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:01.352175 17212 ts_tablet_manager.cc:1905] T 9b73c379a39449999176d408f32f77ed P 9908bc72150a4c2bb48eee3c99b9e5d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:01.353941 17211 ts_tablet_manager.cc:1905] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 17:49:01.354456 16673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:39677 (error 108)
I20260928 17:49:01.355183 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.355060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:39677 (local address 127.0.0.1:58538)
0928 17:49:01.355107 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:01.355151 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 17:49:01.355315 16673 catalog_manager.cc:4584] TS 83a55beccbc24c3598b7e68d5371c3da (127.3.222.1:39677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b73c379a39449999176d408f32f77ed: Network error: Client connection negotiation failed: client connection to 127.3.222.1:39677: connect: Connection refused (error 111)
I20260928 17:49:01.355482 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 16891
I20260928 17:49:01.355818 17211 ts_tablet_manager.cc:1918] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:49:01.355952 17211 log.cc:1196] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/9b73c379a39449999176d408f32f77ed
I20260928 17:49:01.356261 17211 ts_tablet_manager.cc:1939] T 9b73c379a39449999176d408f32f77ed P e34f283b310d4fd689877e9f061bde7b: Deleting consensus metadata
I20260928 17:49:01.356930 16676 catalog_manager.cc:4841] TS e34f283b310d4fd689877e9f061bde7b (127.3.222.3:35859): tablet 9b73c379a39449999176d408f32f77ed (table test-table [id=4d493ecd7d6342ff8c783b1b883f6f6a]) successfully deleted
I20260928 17:49:01.360316 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 17024
I20260928 17:49:01.362798 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:01.361020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:44253 (local address 127.0.0.1:52794)
0928 17:49:01.362733 (+ 1713us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:01.362766 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.2:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1688}
I20260928 17:49:01.364773 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 16656
I20260928 17:49:01.369417 3960 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 17:49:01.713263 3960 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 17:49:01.713354 3960 mini_ranger.cc:66] Stopping Ranger...
I20260928 17:49:02.047312 3960 mini_ranger.cc:68] Stopped Ranger
2026-09-28 17:49:02.047 UTC [15724] LOG: received smart shutdown request
2026-09-28 17:49:02.050 UTC [15724] LOG: background worker "logical replication launcher" (PID 15733) exited with exit code 1
2026-09-28 17:49:02.051 UTC [15728] LOG: shutting down
2026-09-28 17:49:02.215 UTC [15724] LOG: database system is shut down
2026-09-28T17:49:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45543 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:49:02 dist-test-slave-w3bm krb5kdc[17223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:49:02 dist-test-slave-w3bm krb5kdc[17223](info): set up 2 sockets
Sep 28 17:49:02 dist-test-slave-w3bm krb5kdc[17223](info): commencing operation
krb5kdc: starting...
W20260928 17:49:03.246774 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:49:03.360406 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:46641
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:46641,127.3.222.61:34057,127.3.222.60:33471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:03.437500 17236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:03.437693 17236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:03.437721 17236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:03.438822 17236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:49:03.438859 17236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:03.438877 17236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:03.438894 17236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:49:03.438908 17236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:49:03.440347 17236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:46641,127.3.222.61:34057,127.3.222.60:33471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:46641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:03.440572 17236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:03.440771 17236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:03.445686 17242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:03.445760 17236 server_base.cc:1034] running on GCE node
W20260928 17:49:03.445957 17241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:03.445961 17244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:03.446233 17236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:03.446504 17236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:03.447688 17236 hybrid_clock.cc:648] HybridClock initialized: now 1790617743447656 us; error 44 us; skew 500 ppm
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:03.457906 17236 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:49:03.458364 17236 webserver.cc:460] Webserver started at http://127.3.222.62:34483/ using document root <none> and password file <none>
I20260928 17:49:03.458563 17236 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:03.458603 17236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:03.458700 17236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:03.459779 17236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "76af318a45194c66a3d50a2782ea2528"
format_stamp: "Formatted at 2026-09-28 17:49:03 on dist-test-slave-w3bm"
server_key: "cc1f3e1e04466d1f342e97fdb0e0af84"
server_key_iv: "6947767ba4356c1aa46973217f1df273"
server_key_version: "encryptionkey@0"
I20260928 17:49:03.460060 17236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "76af318a45194c66a3d50a2782ea2528"
format_stamp: "Formatted at 2026-09-28 17:49:03 on dist-test-slave-w3bm"
server_key: "cc1f3e1e04466d1f342e97fdb0e0af84"
server_key_iv: "6947767ba4356c1aa46973217f1df273"
server_key_version: "encryptionkey@0"
I20260928 17:49:03.461350 17236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:49:03.461972 17251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:03.462122 17236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:03.462200 17236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "76af318a45194c66a3d50a2782ea2528"
format_stamp: "Formatted at 2026-09-28 17:49:03 on dist-test-slave-w3bm"
server_key: "cc1f3e1e04466d1f342e97fdb0e0af84"
server_key_iv: "6947767ba4356c1aa46973217f1df273"
server_key_version: "encryptionkey@0"
I20260928 17:49:03.462266 17236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:03.475044 17236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:03.477092 17236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:03.477217 17236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:03.480985 17236 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:46641
I20260928 17:49:03.481017 17303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:46641 every 8 connection(s)
I20260928 17:49:03.481335 17236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:49:03.482122 17304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:03.483741 17304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:49:03.484622 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 17236
I20260928 17:49:03.484719 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:49:03.484905 3960 external_mini_cluster.cc:1428] Setting key e63514342e6c47351e04bdd79aca85ae
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.222.61@KRBTEST.COM; defaulting to no policy
I20260928 17:49:03.493196 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.484438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46641 (local address 127.0.0.1:51686)
0928 17:49:03.484628 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.484676 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:03.485274 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:03.485530 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:03.485533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:03.485850 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:03.486060 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.486065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.487025 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.487026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.487746 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.487749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.488260 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.488261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.488323 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.488747 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:03.488754 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:03.490323 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:03.491946 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.491952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.491954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.492133 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.492315 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.492316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.492316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.492381 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.492700 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:03.492706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:03.492784 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 17:49:03.492868 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:49:03.492991 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260928 17:49:03.493211 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.484593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.3.222.62:46641)
0928 17:49:03.484714 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.484719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.484758 (+ 39us) server_negotiation.cc:407] Connection header received
0928 17:49:03.485413 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.485415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.485431 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.485503 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.486148 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.486941 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.487818 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.488189 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.488282 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.490399 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.490406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.490406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.490412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.491877 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.492191 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.492193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.492193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.492269 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.492423 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.492424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.492424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.492623 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.492680 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.492908 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.493042 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":84,"thread_start_us":43,"threads_started":1}
I20260928 17:49:03.493849 17304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:49:03.494140 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.494047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:34057 (local address 127.0.0.1:32768)
0928 17:49:03.494085 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.494124 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 17:49:03.494268 17253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.61@KRBTEST.COM" created.
I20260928 17:49:03.494625 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.494545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:34057 (local address 127.0.0.1:32778)
0928 17:49:03.494581 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.494605 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 17:49:03.494715 17304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.61:34057: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab.
Entry for principal kudu/127.3.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab.
I20260928 17:49:03.509591 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.61:34057
--webserver_interface=127.3.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab
--principal=kudu/127.3.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:46641,127.3.222.61:34057,127.3.222.60:33471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:03.543919 17304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } attempt: 1
I20260928 17:49:03.544377 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.544229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:34057 (local address 127.0.0.1:32784)
0928 17:49:03.544291 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.544338 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 17:49:03.544764 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.544676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:34057 (local address 127.0.0.1:32790)
0928 17:49:03.544716 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.544743 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 17:49:03.544914 17304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.61:34057: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
W20260928 17:49:03.586313 17311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:03.586495 17311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:03.586514 17311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:03.587607 17311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:49:03.587643 17311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:03.587656 17311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:03.587668 17311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:49:03.587692 17311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:49:03.589102 17311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:46641,127.3.222.61:34057,127.3.222.60:33471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab
--principal=kudu/127.3.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.61:34057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:03.589304 17311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:03.589486 17311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:03.593871 17319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:03.593976 17311 server_base.cc:1034] running on GCE node
W20260928 17:49:03.593871 17316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:03.593874 17317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:03.594285 17311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:03.594525 17311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:03.595652 17311 hybrid_clock.cc:648] HybridClock initialized: now 1790617743595641 us; error 31 us; skew 500 ppm
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:03.597991 17311 init.cc:377] Logged in from keytab as kudu/127.3.222.61@KRBTEST.COM (short username kudu)
I20260928 17:49:03.598361 17311 webserver.cc:460] Webserver started at http://127.3.222.61:33775/ using document root <none> and password file <none>
I20260928 17:49:03.598541 17311 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:03.598585 17311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:03.598704 17311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:03.599720 17311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/data/instance:
uuid: "46642ad4863a4edcb4878ee0bad62812"
format_stamp: "Formatted at 2026-09-28 17:49:03 on dist-test-slave-w3bm"
server_key: "d6406091042fc5b273a6b8bc6a4f930e"
server_key_iv: "0e53483b8c1026f0298663db90557767"
server_key_version: "encryptionkey@0"
I20260928 17:49:03.600033 17311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/wal/instance:
uuid: "46642ad4863a4edcb4878ee0bad62812"
format_stamp: "Formatted at 2026-09-28 17:49:03 on dist-test-slave-w3bm"
server_key: "d6406091042fc5b273a6b8bc6a4f930e"
server_key_iv: "0e53483b8c1026f0298663db90557767"
server_key_version: "encryptionkey@0"
I20260928 17:49:03.601212 17311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:49:03.601866 17326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:03.602046 17311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:03.602123 17311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/wal
uuid: "46642ad4863a4edcb4878ee0bad62812"
format_stamp: "Formatted at 2026-09-28 17:49:03 on dist-test-slave-w3bm"
server_key: "d6406091042fc5b273a6b8bc6a4f930e"
server_key_iv: "0e53483b8c1026f0298663db90557767"
server_key_version: "encryptionkey@0"
I20260928 17:49:03.602205 17311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:03.611166 17311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:03.612830 17311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:03.612950 17311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:03.613073 17304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } attempt: 2
I20260928 17:49:03.613463 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.613317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:34057 (local address 127.0.0.1:32798)
0928 17:49:03.613381 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.613423 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 17:49:03.613801 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.613715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:34057 (local address 127.0.0.1:32814)
0928 17:49:03.613755 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.613781 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 17:49:03.613883 17304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.61:34057: Network error: Client connection negotiation failed: client connection to 127.3.222.61:34057: connect: Connection refused (error 111)
I20260928 17:49:03.616727 17311 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.61:34057
I20260928 17:49:03.616772 17378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.61:34057 every 8 connection(s)
I20260928 17:49:03.617045 17311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
I20260928 17:49:03.617789 17379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:03.619274 17379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:49:03.623728 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 17311
I20260928 17:49:03.623823 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-1/wal/instance
I20260928 17:49:03.624006 3960 external_mini_cluster.cc:1428] Setting key fc6a4abb2e05ef98598c92964065b924
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:03.627385 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.619873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.3.222.62:46641)
0928 17:49:03.619936 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.619939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.620106 (+ 167us) server_negotiation.cc:407] Connection header received
0928 17:49:03.620553 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.620553 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.620565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.620602 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.621144 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.621628 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.622548 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.622742 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.622787 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.624886 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.624887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.624888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.624894 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.626085 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.626493 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.626494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.626494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.626550 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.626782 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.626783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.626783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.626891 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.626940 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.627251 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.627326 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:49:03.627589 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.619847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46641 (local address 127.0.0.1:51702)
0928 17:49:03.620002 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.620035 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:03.620470 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:03.620668 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:03.620671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:03.620889 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:03.621066 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.621070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.621701 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.621701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.622481 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.622483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.622799 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.622800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.622899 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.623372 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:03.623380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:03.624802 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:03.626158 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.626166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.626168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.626428 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.626609 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.626610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.626610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.626725 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.626943 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:03.626948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:03.627088 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 17:49:03.627201 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 17:49:03.627376 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260928 17:49:03.628060 17379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.60@KRBTEST.COM" created.
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:49:03.637271 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.628292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:34057 (local address 127.0.0.1:32824)
0928 17:49:03.628469 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.628480 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:03.628535 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:03.628688 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:03.628689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:03.628819 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:03.628872 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.628874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.629653 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.629654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.630413 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.630415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.630870 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.630871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.630941 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.631290 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:03.631295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:03.634925 (+ 3630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:03.636393 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.636394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.636395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.636560 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.636760 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.636760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.636760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.636803 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.637052 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:03.637052 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:03.637083 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 17:49:03.637126 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:49:03.637210 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 17:49:03.637384 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.628348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.3.222.61:34057)
0928 17:49:03.628476 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.628480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.628529 (+ 49us) server_negotiation.cc:407] Connection header received
0928 17:49:03.628587 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.628588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.628598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.628674 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.628933 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.629577 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.630493 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.630800 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.630853 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.634998 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.635005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.635006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.635011 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.636318 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.636642 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.636646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.636646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.636704 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.636847 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.636847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.636848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.637002 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.637073 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.637167 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.637270 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260928 17:49:03.638000 17379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:49:03.638376 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.638256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42784)
0928 17:49:03.638304 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.638352 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 17:49:03.638501 17327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
I20260928 17:49:03.638898 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.638810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42788)
0928 17:49:03.638850 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.638877 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 17:49:03.638991 17379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:33471: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab.
Entry for principal kudu/127.3.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab.
I20260928 17:49:03.649617 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.60:33471
--webserver_interface=127.3.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab
--principal=kudu/127.3.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:46641,127.3.222.61:34057,127.3.222.60:33471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:03.688206 17379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } attempt: 1
I20260928 17:49:03.688714 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.688570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42790)
0928 17:49:03.688634 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.688679 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 17:49:03.689031 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.688962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42792)
0928 17:49:03.688994 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.689016 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 17:49:03.689118 17379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:33471: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
I20260928 17:49:03.705008 17304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } attempt: 3
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:49:03.711879 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.705401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.3.222.61:34057)
0928 17:49:03.705473 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.705475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.705488 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:49:03.705563 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.705564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.705575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.705633 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.705913 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.706688 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.707347 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.707690 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.707758 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.709556 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.709557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.709557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.709564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.710735 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.711165 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.711166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.711166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.711269 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.711462 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.711463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.711464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.711566 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.711613 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.711751 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.711814 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:49:03.711934 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.705365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:34057 (local address 127.0.0.1:32826)
0928 17:49:03.705428 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.705440 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:03.705496 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:03.705643 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:03.705644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:03.705787 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:03.705856 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.705858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.706756 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.706757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.707287 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.707288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.707751 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.707751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.707804 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.708245 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:03.708249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:03.709461 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:03.710821 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.710822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.710823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.711056 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.711332 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.711333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.711334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.711404 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.711615 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:03.711615 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:03.711668 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 17:49:03.711736 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:49:03.711837 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 17:49:03.712321 17304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:49:03.712654 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.712538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42806)
0928 17:49:03.712594 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.712630 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 17:49:03.712985 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.712880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42814)
0928 17:49:03.712933 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.712966 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 17:49:03.713073 17304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:33471: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
W20260928 17:49:03.731242 17386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:03.731447 17386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:03.731468 17386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:03.732559 17386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:49:03.732591 17386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:03.732604 17386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:03.732614 17386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:49:03.732625 17386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:49:03.734023 17386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:46641,127.3.222.61:34057,127.3.222.60:33471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab
--principal=kudu/127.3.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.60:33471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:03.734208 17386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:03.734378 17386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:03.738775 17391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:03.738873 17386 server_base.cc:1034] running on GCE node
W20260928 17:49:03.738937 17394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:03.738782 17392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:03.739250 17386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:03.739569 17386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:03.740720 17386 hybrid_clock.cc:648] HybridClock initialized: now 1790617743740680 us; error 31 us; skew 500 ppm
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:03.742966 17386 init.cc:377] Logged in from keytab as kudu/127.3.222.60@KRBTEST.COM (short username kudu)
I20260928 17:49:03.743417 17386 webserver.cc:460] Webserver started at http://127.3.222.60:45381/ using document root <none> and password file <none>
I20260928 17:49:03.743613 17386 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:03.743654 17386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:03.743817 17386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:03.744236 17304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } attempt: 1
I20260928 17:49:03.744611 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.744494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42830)
0928 17:49:03.744549 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.744583 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:49:03.744915 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.744846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42840)
0928 17:49:03.744869 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.744894 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 17:49:03.745025 17304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:33471: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
I20260928 17:49:03.745148 17386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/data/instance:
uuid: "78d62e0ec58b4f9396485596b9d54677"
format_stamp: "Formatted at 2026-09-28 17:49:03 on dist-test-slave-w3bm"
server_key: "991dcf30b09ebf47a4e3756e831043f5"
server_key_iv: "062dcaf9a094bc036e361208982f3417"
server_key_version: "encryptionkey@0"
I20260928 17:49:03.745463 17386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/wal/instance:
uuid: "78d62e0ec58b4f9396485596b9d54677"
format_stamp: "Formatted at 2026-09-28 17:49:03 on dist-test-slave-w3bm"
server_key: "991dcf30b09ebf47a4e3756e831043f5"
server_key_iv: "062dcaf9a094bc036e361208982f3417"
server_key_version: "encryptionkey@0"
I20260928 17:49:03.746651 17386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:49:03.747313 17401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:03.747449 17386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:03.747514 17386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/wal
uuid: "78d62e0ec58b4f9396485596b9d54677"
format_stamp: "Formatted at 2026-09-28 17:49:03 on dist-test-slave-w3bm"
server_key: "991dcf30b09ebf47a4e3756e831043f5"
server_key_iv: "062dcaf9a094bc036e361208982f3417"
server_key_version: "encryptionkey@0"
I20260928 17:49:03.747579 17386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:03.757261 17379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } attempt: 2
I20260928 17:49:03.757633 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.757525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42850)
0928 17:49:03.757577 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.757605 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 17:49:03.757942 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.757847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42856)
0928 17:49:03.757887 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.757919 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 17:49:03.758040 17379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:33471: Network error: Client connection negotiation failed: client connection to 127.3.222.60:33471: connect: Connection refused (error 111)
I20260928 17:49:03.766752 17386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:03.768738 17386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:03.768863 17386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:03.772532 17386 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.60:33471
I20260928 17:49:03.772575 17453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.60:33471 every 8 connection(s)
I20260928 17:49:03.772883 17386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
I20260928 17:49:03.773682 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:03.773736 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 17386
I20260928 17:49:03.773814 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617599653609-3960-0/minicluster-data/master-2/wal/instance
I20260928 17:49:03.773996 3960 external_mini_cluster.cc:1428] Setting key b337e51a9ab4956d8ec95f44a93a69df
I20260928 17:49:03.775940 17454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:03.783516 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.774480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.3.222.60:33471)
0928 17:49:03.774617 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.774625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.774650 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:49:03.775098 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.775105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.775160 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.775301 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.775982 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.777054 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.777813 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.778248 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.778323 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.780730 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.780737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.780737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.780744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.782081 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.782384 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.782387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.782388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.782468 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.782659 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.782660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.782660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.782917 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.783045 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.783170 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.783308 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260928 17:49:03.783645 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.776561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.3.222.62:46641)
0928 17:49:03.776614 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.776616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.776763 (+ 147us) server_negotiation.cc:407] Connection header received
0928 17:49:03.776823 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.776824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.776833 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.776876 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.777333 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.777784 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.778575 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.778863 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.778916 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.781187 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.781188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.781188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.781195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.782450 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.782811 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.782812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.782812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.782866 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.783055 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.783056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.783056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.783193 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.783271 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.783489 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.783563 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:49:03.783687 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.776521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:46641 (local address 127.0.0.1:51712)
0928 17:49:03.776666 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.776703 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:03.776765 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:03.776898 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:03.776901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:03.777169 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:03.777254 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.777257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.777867 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.777868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.778499 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.778501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.778937 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.778938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.779010 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.779399 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:03.779409 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:03.781115 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:03.782512 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.782517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.782519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.782732 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.782929 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.782929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.782930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.782985 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.783302 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:03.783309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:03.783361 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 17:49:03.783436 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:49:03.783543 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260928 17:49:03.784160 17454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:03.790239 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.784409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:34057 (local address 127.0.0.1:32836)
0928 17:49:03.784453 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.784462 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:03.784493 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:03.784648 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:03.784649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:03.784757 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:03.784785 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.784786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.785440 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.785441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.785957 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.785959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.786323 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.786324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.786374 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.786720 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:03.786724 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:03.787894 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:03.789400 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.789401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.789402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.789585 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.789773 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.789773 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.789774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.789834 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.790023 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:03.790023 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:03.790063 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 17:49:03.790117 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:49:03.790180 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:49:03.790297 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.784482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.3.222.61:34057)
0928 17:49:03.784550 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.784552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.784560 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:49:03.784582 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.784583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.784592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.784643 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.784842 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.785361 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.786022 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.786270 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.786314 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.787986 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.787987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.787987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.787994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.789327 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.789671 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.789671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.789672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.789719 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.789883 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.789883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.789884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.789971 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.790025 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.790148 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.790210 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:49:03.790917 17454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:49:03.791072 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.784292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.3.222.62:46641)
0928 17:49:03.784327 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.784329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.784338 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:49:03.784384 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.784385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.784394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.784515 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.785041 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.785522 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.786161 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.786346 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.786404 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.788560 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.788562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.788562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.788568 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.789949 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.790286 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.790287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.790287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.790344 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.790529 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.790530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.790531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.790633 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.790885 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.790923 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.790989 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:49:03.797220 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.791148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42878)
0928 17:49:03.791181 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.791190 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:03.791258 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:03.791385 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:03.791386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:03.791503 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:03.791529 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.791530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.792207 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.792208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.792700 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.792702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.793011 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.793011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.793063 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.793410 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:03.793415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:03.794571 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:03.796413 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.796414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.796414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.796557 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.796713 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.796714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.796714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.796764 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.796950 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:03.796951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:03.796985 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 17:49:03.797042 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:49:03.797140 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 17:49:03.797521 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.791234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.3.222.60:33471)
0928 17:49:03.791280 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.791283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.791291 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:49:03.791308 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.791309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.791318 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.791373 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.791579 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.792141 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.792768 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.792958 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.793002 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.794696 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.794698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.794698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.794704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.796349 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.796611 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.796612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.796612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.796666 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.796810 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.796811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.796811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.796904 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.796953 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.797070 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.797472 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":338,"server-negotiator.queue_time_us":27}
I20260928 17:49:03.798096 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.791751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.3.222.61:34057)
0928 17:49:03.791786 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.791787 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.791794 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:49:03.791811 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.791812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.791820 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.791876 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.792253 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.792684 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.793415 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.793609 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.793664 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.795853 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.795854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.795855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.795860 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.797104 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.797412 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.797413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.797413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.797459 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.797665 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.797666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.797667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.797784 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.797924 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.797988 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.798042 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:49:03.798861 17454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677: Bootstrap starting.
I20260928 17:49:03.799367 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:49:03.799710 17454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:03.799948 17454 log.cc:824] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:03.800724 17454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677: No bootstrap required, opened a new log
I20260928 17:49:03.802700 17454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } }
I20260928 17:49:03.802836 17454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:03.802865 17454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d62e0ec58b4f9396485596b9d54677, State: Initialized, Role: FOLLOWER
I20260928 17:49:03.802979 17454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } }
I20260928 17:49:03.803299 17454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:49:03.803257 17466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:03.803349 17466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:49:03.805866 17477 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:49:03.805915 17477 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:49:03.806809 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.800651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.3.222.61:34057)
0928 17:49:03.800686 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.800687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.800788 (+ 101us) server_negotiation.cc:407] Connection header received
0928 17:49:03.800839 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.800840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.800849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.800898 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.801377 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.801810 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.802429 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.802625 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.802674 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.803799 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.803800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.803800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.803807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.805214 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.805560 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.805561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.805562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.805614 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.805794 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.805795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.805796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.805910 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.806441 (+ 531us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.806653 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.806723 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:49:03.808347 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.800633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.3.222.62:46641)
0928 17:49:03.800670 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.800671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.800890 (+ 219us) server_negotiation.cc:407] Connection header received
0928 17:49:03.800966 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.800967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.800976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.801024 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.801573 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.802235 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.803193 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.803485 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.803528 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.804837 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.804839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.804839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.804845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.806254 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.807420 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.807421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.807421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.807480 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.807735 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.807736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.807737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.807896 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.807999 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.808183 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.808261 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:49:03.806756 17386 master_runner.cc:186] Error getting master registration for 127.3.222.62:46641: OK, instance_id { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" instance_seqno: 1790617743479951 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:49:03.809484 17386 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:49:03.811627 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.804961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.3.222.60:33471)
0928 17:49:03.805058 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.805060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.805326 (+ 266us) server_negotiation.cc:407] Connection header received
0928 17:49:03.805456 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.805469 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.805477 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.805876 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.806328 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.806909 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.807959 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.808163 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.808217 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.809322 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.809323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.809323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.809330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.810604 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.810939 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.810940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.810940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.810995 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.811128 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.811128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.811128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.811300 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.811382 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.811479 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.811548 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 17:49:03.820160 17304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } attempt: 2
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
I20260928 17:49:03.826220 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.820413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42904)
0928 17:49:03.820463 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.820472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:03.820508 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:03.820632 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:03.820633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:03.820800 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:03.820841 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.820843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.821419 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.821420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.821898 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.821899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.822258 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.822259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.822315 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.822693 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:03.822698 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:03.823823 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:03.825447 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.825448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.825448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.825590 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.825757 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.825757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.825758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.825800 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.825999 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:03.825999 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:03.826032 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 17:49:03.826075 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:49:03.826141 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:49:03.826256 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.820499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.3.222.60:33471)
0928 17:49:03.820547 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.820549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.820555 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:49:03.820573 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.820574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.820581 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.820622 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.820907 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.821353 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.821963 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.822143 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.822207 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.823901 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.823903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.823903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.823910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.825380 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.825645 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.825645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.825646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.825704 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.825834 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.825834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.825835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.825950 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.826007 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.826098 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.826168 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:49:03.827785 17304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528: Bootstrap starting.
I20260928 17:49:03.828366 17304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:03.828608 17304 log.cc:824] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:03.829284 17304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528: No bootstrap required, opened a new log
I20260928 17:49:03.830952 17304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } }
I20260928 17:49:03.831094 17304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:03.831130 17304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76af318a45194c66a3d50a2782ea2528, State: Initialized, Role: FOLLOWER
I20260928 17:49:03.831310 17304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } }
I20260928 17:49:03.831564 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:03.831633 17304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:49:03.831642 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:49:03.833175 17492 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:49:03.833227 17492 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:49:03.833930 17236 master_runner.cc:186] Error getting master registration for 127.3.222.61:34057: OK, instance_id { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" instance_seqno: 1790617743615715 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:49:03.836308 17236 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:49:03.849195 17379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } attempt: 3
Sep 28 17:49:03 dist-test-slave-w3bm krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617743, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
I20260928 17:49:03.857466 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.849537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.3.222.60:33471)
0928 17:49:03.849593 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:03.849596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:03.849706 (+ 110us) server_negotiation.cc:407] Connection header received
0928 17:49:03.849729 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:03.849730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:03.849740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:03.849788 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:03.850153 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.850888 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.851913 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.852210 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.852276 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.854753 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:03.854754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:03.854755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:03.854762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:03.856355 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.856730 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.856731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.856731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.856789 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:03.856973 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:03.856974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:03.856974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:03.857102 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:03.857163 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:03.857300 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 17:49:03.857372 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:49:03.857462 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:03.849482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:33471 (local address 127.0.0.1:42918)
0928 17:49:03.849535 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:03.849606 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:03.849654 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:03.849814 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:03.849815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:03.850017 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:03.850069 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.850071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.850985 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.850986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.851837 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:03.851840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:03.852623 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:03.852624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:03.852691 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:03.853174 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:03.853180 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:03.854656 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:03.856437 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.856438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.856439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.856651 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.856853 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:03.856853 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:03.856854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:03.856917 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:03.857159 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:03.857159 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:03.857206 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 17:49:03.857271 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 17:49:03.857362 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 17:49:03.859727 17379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812: Bootstrap starting.
I20260928 17:49:03.860430 17379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:03.860740 17379 log.cc:824] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:03.861568 17379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812: No bootstrap required, opened a new log
I20260928 17:49:03.863615 17379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } }
I20260928 17:49:03.863767 17379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:03.863799 17379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46642ad4863a4edcb4878ee0bad62812, State: Initialized, Role: FOLLOWER
I20260928 17:49:03.864439 17379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } }
I20260928 17:49:03.864791 17379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 17:49:03.866647 17506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:49:03.866747 17506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:49:03.868556 17311 master_runner.cc:263] No leader master found from master 46642ad4863a4edcb4878ee0bad62812
I20260928 17:49:03.868607 17311 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:49:03.871335 17495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:03.871429 17495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:49:03.977993 17257 tablet.cc:2369] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:49:04.019462 17407 tablet.cc:2369] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:49:04.035197 17481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:49:04.035327 17481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } }
I20260928 17:49:04.035547 17481 leader_election.cc:288] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46642ad4863a4edcb4878ee0bad62812 (127.3.222.61:34057), 78d62e0ec58b4f9396485596b9d54677 (127.3.222.60:33471)
I20260928 17:49:04.035900 17353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76af318a45194c66a3d50a2782ea2528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46642ad4863a4edcb4878ee0bad62812" is_pre_election: true
I20260928 17:49:04.035887 17428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76af318a45194c66a3d50a2782ea2528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d62e0ec58b4f9396485596b9d54677" is_pre_election: true
I20260928 17:49:04.036054 17428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76af318a45194c66a3d50a2782ea2528 in term 0.
I20260928 17:49:04.036054 17353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76af318a45194c66a3d50a2782ea2528 in term 0.
I20260928 17:49:04.036245 17252 leader_election.cc:302] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76af318a45194c66a3d50a2782ea2528, 78d62e0ec58b4f9396485596b9d54677; no voters:
I20260928 17:49:04.036398 17481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:49:04.036463 17481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:49:04.036478 17481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:04.037207 17481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } }
I20260928 17:49:04.037324 17481 leader_election.cc:288] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [CANDIDATE]: Term 1 election: Requested vote from peers 46642ad4863a4edcb4878ee0bad62812 (127.3.222.61:34057), 78d62e0ec58b4f9396485596b9d54677 (127.3.222.60:33471)
I20260928 17:49:04.037467 17353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76af318a45194c66a3d50a2782ea2528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46642ad4863a4edcb4878ee0bad62812"
I20260928 17:49:04.037483 17428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76af318a45194c66a3d50a2782ea2528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d62e0ec58b4f9396485596b9d54677"
I20260928 17:49:04.037544 17353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:04.037568 17428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:04.038226 17353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76af318a45194c66a3d50a2782ea2528 in term 1.
I20260928 17:49:04.038317 17428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76af318a45194c66a3d50a2782ea2528 in term 1.
I20260928 17:49:04.038424 17253 leader_election.cc:302] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46642ad4863a4edcb4878ee0bad62812, 76af318a45194c66a3d50a2782ea2528; no voters:
I20260928 17:49:04.038533 17481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:49:04.038673 17481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [term 1 LEADER]: Becoming Leader. State: Replica: 76af318a45194c66a3d50a2782ea2528, State: Running, Role: LEADER
I20260928 17:49:04.038765 17481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } }
I20260928 17:49:04.039161 17508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76af318a45194c66a3d50a2782ea2528. Latest consensus state: current_term: 1 leader_uuid: "76af318a45194c66a3d50a2782ea2528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:04.039290 17508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:04.039489 17510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:49:04.039656 17510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:49:04.040938 17353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [term 1 FOLLOWER]: Refusing update from remote peer 76af318a45194c66a3d50a2782ea2528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:04.040995 17428 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [term 1 FOLLOWER]: Refusing update from remote peer 76af318a45194c66a3d50a2782ea2528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:04.041278 17508 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:04.041397 17481 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:04.042483 17466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76af318a45194c66a3d50a2782ea2528. Latest consensus state: current_term: 1 leader_uuid: "76af318a45194c66a3d50a2782ea2528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:04.042502 17495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76af318a45194c66a3d50a2782ea2528. Latest consensus state: current_term: 1 leader_uuid: "76af318a45194c66a3d50a2782ea2528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:04.042570 17466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:49:04.042578 17495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:49:04.043380 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76af318a45194c66a3d50a2782ea2528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:04.043458 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:04.043540 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76af318a45194c66a3d50a2782ea2528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:04.043592 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528 [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:04.043622 17466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76af318a45194c66a3d50a2782ea2528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:04.043696 17466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:49:04.043702 17512 mvcc.cc:205] Tried to move back new op lower bound from 7334370279589617664 to 7334370279583428608. Current Snapshot: MvccSnapshot[applied={T|T < 7334370279589617664}]
I20260928 17:49:04.044032 17510 catalog_manager.cc:1347] Generated new cluster ID: 82f739d09cce44babf509aca6e57bd04
I20260928 17:49:04.044070 17510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:49:04.044021 17495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76af318a45194c66a3d50a2782ea2528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76af318a45194c66a3d50a2782ea2528" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 46641 } } peers { permanent_uuid: "46642ad4863a4edcb4878ee0bad62812" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 34057 } } peers { permanent_uuid: "78d62e0ec58b4f9396485596b9d54677" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 33471 } } }
I20260928 17:49:04.044094 17495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:49:04.050879 17510 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:49:04.051569 17510 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:49:04.059508 17510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76af318a45194c66a3d50a2782ea2528: Generated new TSK 0
I20260928 17:49:04.059692 17510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:49:04.806347 17477 catalog_manager.cc:1259] Loaded cluster ID: 82f739d09cce44babf509aca6e57bd04
I20260928 17:49:04.806420 17477 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677: loading cluster ID for follower catalog manager: success
I20260928 17:49:04.807204 17477 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677: acquiring CA information for follower catalog manager: success
I20260928 17:49:04.807516 17477 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78d62e0ec58b4f9396485596b9d54677: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:49:04.867184 17506 catalog_manager.cc:1259] Loaded cluster ID: 82f739d09cce44babf509aca6e57bd04
I20260928 17:49:04.867285 17506 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812: loading cluster ID for follower catalog manager: success
I20260928 17:49:04.868083 17506 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812: acquiring CA information for follower catalog manager: success
I20260928 17:49:04.868371 17506 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46642ad4863a4edcb4878ee0bad62812: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:49:05.106006 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 17236
I20260928 17:49:05.111780 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 17311
I20260928 17:49:05.115938 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 17386
2026-09-28T17:49:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2813 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:49:05 dist-test-slave-w3bm krb5kdc[17525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:49:05 dist-test-slave-w3bm krb5kdc[17525](info): set up 2 sockets
Sep 28 17:49:05 dist-test-slave-w3bm krb5kdc[17525](info): commencing operation
krb5kdc: starting...
W20260928 17:49:06.063002 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:49:06 dist-test-slave-w3bm krb5kdc[17525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:49:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:49:06.174779 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:37307
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:37307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:06.251338 17538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:06.251531 17538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:06.251559 17538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:06.252674 17538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:49:06.252719 17538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:06.252738 17538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:06.252754 17538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:49:06.252780 17538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:49:06.254205 17538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:37307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:37307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:06.254428 17538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:06.254611 17538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:06.260118 17544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:06.260133 17543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:06.260190 17546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:06.260849 17538 server_base.cc:1034] running on GCE node
I20260928 17:49:06.261113 17538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:06.261461 17538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:06.262624 17538 hybrid_clock.cc:648] HybridClock initialized: now 1790617746262609 us; error 37 us; skew 500 ppm
Sep 28 17:49:06 dist-test-slave-w3bm krb5kdc[17525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617746, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:06.265322 17538 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:49:06.265718 17538 webserver.cc:460] Webserver started at http://127.3.222.62:34223/ using document root <none> and password file <none>
I20260928 17:49:06.265906 17538 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:06.265951 17538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:06.266069 17538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:06.267030 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "8db929675c154f81956bd5b636584dbc"
format_stamp: "Formatted at 2026-09-28 17:49:06 on dist-test-slave-w3bm"
server_key: "4f1a2680c2534366a6d2cce8d5dc1e75"
server_key_iv: "bd3a1b56532f406ce1808728dd87e304"
server_key_version: "encryptionkey@0"
I20260928 17:49:06.267364 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "8db929675c154f81956bd5b636584dbc"
format_stamp: "Formatted at 2026-09-28 17:49:06 on dist-test-slave-w3bm"
server_key: "4f1a2680c2534366a6d2cce8d5dc1e75"
server_key_iv: "bd3a1b56532f406ce1808728dd87e304"
server_key_version: "encryptionkey@0"
I20260928 17:49:06.268484 17538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:49:06.269083 17553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:06.269236 17538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:49:06.269312 17538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "8db929675c154f81956bd5b636584dbc"
format_stamp: "Formatted at 2026-09-28 17:49:06 on dist-test-slave-w3bm"
server_key: "4f1a2680c2534366a6d2cce8d5dc1e75"
server_key_iv: "bd3a1b56532f406ce1808728dd87e304"
server_key_version: "encryptionkey@0"
I20260928 17:49:06.269375 17538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:06.292426 17538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:06.294363 17538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:06.294497 17538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:06.298085 17538 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:37307
I20260928 17:49:06.298131 17605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:37307 every 8 connection(s)
I20260928 17:49:06.298434 17538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:49:06.298647 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 17538
I20260928 17:49:06.298743 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:49:06.298951 3960 external_mini_cluster.cc:1428] Setting key 65300caae879694c8cf8e6c2fff6345f
I20260928 17:49:06.299412 17606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:06.302239 17606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc: Bootstrap starting.
I20260928 17:49:06.302922 17606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:06.303187 17606 log.cc:824] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:06.303983 17606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc: No bootstrap required, opened a new log
Sep 28 17:49:06 dist-test-slave-w3bm krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:06.306327 17606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db929675c154f81956bd5b636584dbc" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 37307 } }
I20260928 17:49:06.306504 17606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:06.306542 17606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db929675c154f81956bd5b636584dbc, State: Initialized, Role: FOLLOWER
I20260928 17:49:06.306676 17606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db929675c154f81956bd5b636584dbc" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 37307 } }
I20260928 17:49:06.306749 17606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:49:06.306771 17606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:49:06.306815 17606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:06.307374 17606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db929675c154f81956bd5b636584dbc" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 37307 } }
I20260928 17:49:06.307468 17606 leader_election.cc:302] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8db929675c154f81956bd5b636584dbc; no voters:
I20260928 17:49:06.307669 17606 leader_election.cc:288] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:49:06.307709 17611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:49:06.307816 17611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [term 1 LEADER]: Becoming Leader. State: Replica: 8db929675c154f81956bd5b636584dbc, State: Running, Role: LEADER
I20260928 17:49:06.307928 17606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:49:06.307957 17611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db929675c154f81956bd5b636584dbc" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 37307 } }
I20260928 17:49:06.308302 17611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db929675c154f81956bd5b636584dbc. Latest consensus state: current_term: 1 leader_uuid: "8db929675c154f81956bd5b636584dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db929675c154f81956bd5b636584dbc" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 37307 } } }
I20260928 17:49:06.308367 17611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:06.308774 17612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8db929675c154f81956bd5b636584dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db929675c154f81956bd5b636584dbc" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 37307 } } }
I20260928 17:49:06.308842 17612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:06.308980 17616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:49:06.309437 17616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:49:06.310024 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:06.299424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.3.222.62:37307)
0928 17:49:06.299590 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:06.299600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:06.299627 (+ 27us) server_negotiation.cc:407] Connection header received
0928 17:49:06.300052 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:06.300058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:06.300102 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:06.300219 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:06.300966 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:06.301879 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:06.302910 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:06.303449 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:06.303570 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:06.306006 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:06.306009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:06.306010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:06.306020 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:06.307922 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:06.308306 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:06.308308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:06.308308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:06.308438 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:06.308675 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:06.308675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:06.308676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:06.308978 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:06.309495 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:06.309589 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 17:49:06.309735 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260928 17:49:06.310916 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:49:06.311287 17616 catalog_manager.cc:1347] Generated new cluster ID: 403afb8d5b37472d99101ae282970c8b
I20260928 17:49:06.311336 17616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:49:06.342463 17616 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:49:06.343200 17616 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:49:06.346861 17616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8db929675c154f81956bd5b636584dbc: Generated new TSK 0
I20260928 17:49:06.347155 17616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:49:06.364674 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 17538
2026-09-28T17:49:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1247 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:49:06 dist-test-slave-w3bm krb5kdc[17633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:49:06 dist-test-slave-w3bm krb5kdc[17633](info): set up 2 sockets
Sep 28 17:49:06 dist-test-slave-w3bm krb5kdc[17633](info): commencing operation
krb5kdc: starting...
W20260928 17:49:07.298825 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:49:07 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:49:07.406013 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38581
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:38581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:07.482522 17646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:07.482707 17646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:07.482735 17646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:07.483875 17646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:49:07.483912 17646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:07.483932 17646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:07.483947 17646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:49:07.483968 17646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:49:07.485383 17646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:38581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:07.485605 17646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:07.485790 17646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:07.490128 17652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:07.490128 17654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:07.490247 17646 server_base.cc:1034] running on GCE node
W20260928 17:49:07.490190 17651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:07.490542 17646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:07.490792 17646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:07.491925 17646 hybrid_clock.cc:648] HybridClock initialized: now 1790617747491916 us; error 27 us; skew 500 ppm
Sep 28 17:49:07 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:07.494122 17646 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:49:07.494477 17646 webserver.cc:460] Webserver started at http://127.3.222.62:34285/ using document root <none> and password file <none>
I20260928 17:49:07.494653 17646 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:07.494691 17646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:07.494774 17646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:07.495710 17646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "c900a3e55b2447db96c4721ca6f6d142"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "9f4bdfb6a37cc2613a92f1458f9ee9f3"
server_key_iv: "d82e5ef7bb2d2e70ae432950f0b9a7a2"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.495971 17646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "c900a3e55b2447db96c4721ca6f6d142"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "9f4bdfb6a37cc2613a92f1458f9ee9f3"
server_key_iv: "d82e5ef7bb2d2e70ae432950f0b9a7a2"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.497058 17646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:49:07.497633 17661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.497799 17646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:07.497871 17646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "c900a3e55b2447db96c4721ca6f6d142"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "9f4bdfb6a37cc2613a92f1458f9ee9f3"
server_key_iv: "d82e5ef7bb2d2e70ae432950f0b9a7a2"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.497920 17646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:07.510459 17646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:07.512372 17646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:07.512514 17646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:07.515990 17646 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:38581
I20260928 17:49:07.516031 17713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:38581 every 8 connection(s)
I20260928 17:49:07.516323 17646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:49:07.517210 17714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:07.519289 17714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Bootstrap starting.
I20260928 17:49:07.519586 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 17646
I20260928 17:49:07.519672 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:49:07.519831 17714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:07.519845 3960 external_mini_cluster.cc:1428] Setting key b561f59c8956e84b10b8db6fa5b4c3d9
I20260928 17:49:07.520071 17714 log.cc:824] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:07.520727 17714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: No bootstrap required, opened a new log
I20260928 17:49:07.522401 17714 raft_consensus.cc:348] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:07.522526 17714 raft_consensus.cc:374] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:07.522552 17714 raft_consensus.cc:729] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c900a3e55b2447db96c4721ca6f6d142, State: Initialized, Role: FOLLOWER
I20260928 17:49:07.522680 17714 consensus_queue.cc:260] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:07.522768 17714 raft_consensus.cc:388] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:49:07.522804 17714 raft_consensus.cc:482] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:49:07.522847 17714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:07.523509 17714 raft_consensus.cc:504] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:07.523608 17714 leader_election.cc:302] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c900a3e55b2447db96c4721ca6f6d142; no voters:
I20260928 17:49:07.523758 17714 leader_election.cc:288] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:49:07.523778 17719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:49:07.523892 17719 raft_consensus.cc:686] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 1 LEADER]: Becoming Leader. State: Replica: c900a3e55b2447db96c4721ca6f6d142, State: Running, Role: LEADER
I20260928 17:49:07.523993 17714 sys_catalog.cc:564] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:49:07.524024 17719 consensus_queue.cc:237] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:07.524322 17719 sys_catalog.cc:455] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c900a3e55b2447db96c4721ca6f6d142. Latest consensus state: current_term: 1 leader_uuid: "c900a3e55b2447db96c4721ca6f6d142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } } }
I20260928 17:49:07.524394 17719 sys_catalog.cc:458] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:07.524537 17720 sys_catalog.cc:455] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c900a3e55b2447db96c4721ca6f6d142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } } }
I20260928 17:49:07.524596 17720 sys_catalog.cc:458] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: This master's current role is: LEADER
W20260928 17:49:07.525640 17735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:49:07.525842 17735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:49:07.525905 17725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:49:07.526084 17725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 17:49:07 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:07.527804 17725 catalog_manager.cc:1347] Generated new cluster ID: 50eaabda8e124023b332028b8a5fa835
I20260928 17:49:07.527848 17725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:49:07.530270 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.520271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.3.222.62:38581)
0928 17:49:07.520432 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:07.520440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:07.520463 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:49:07.520899 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:07.520906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:07.520952 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:07.521080 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:07.521650 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.522480 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.523539 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.524099 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.524234 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.526749 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:07.526753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:07.526753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:07.526764 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:07.528567 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.528936 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.528939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.528939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.529037 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.529227 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.529228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.529228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.529485 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:07.529624 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:07.529729 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 17:49:07.529849 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260928 17:49:07.537855 17725 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:49:07.538420 17725 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:49:07.541000 17725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Generated new TSK 0
I20260928 17:49:07.541150 17725 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:49:07.567018 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38581
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:07.643419 17740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:07.643595 17740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:07.643622 17740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:07.644692 17740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:07.644721 17740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:07.644745 17740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:49:07.646070 17740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:07.646229 17740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:07.646389 17740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:07.650914 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:07.650938 17740 server_base.cc:1034] running on GCE node
W20260928 17:49:07.651064 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:07.650949 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:07.651378 17740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:07.651564 17740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:07.652686 17740 hybrid_clock.cc:648] HybridClock initialized: now 1790617747652678 us; error 25 us; skew 500 ppm
Sep 28 17:49:07 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:07.654971 17740 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:49:07.655403 17740 webserver.cc:460] Webserver started at http://127.3.222.1:36859/ using document root <none> and password file <none>
I20260928 17:49:07.655588 17740 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:07.655639 17740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:07.655759 17740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:07.656701 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "59b3f406e90e41da95eeea9034e35f2e"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "e1eb72288d0e1a45b012a0718d774d7b"
server_key_iv: "beb253e73d407a121291e39cb67d29bf"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.657007 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "59b3f406e90e41da95eeea9034e35f2e"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "e1eb72288d0e1a45b012a0718d774d7b"
server_key_iv: "beb253e73d407a121291e39cb67d29bf"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.658150 17740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:49:07.658741 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.658890 17740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:07.658965 17740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "59b3f406e90e41da95eeea9034e35f2e"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "e1eb72288d0e1a45b012a0718d774d7b"
server_key_iv: "beb253e73d407a121291e39cb67d29bf"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.659029 17740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:07.666484 17740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:07.668098 17740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:07.668213 17740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:07.668385 17740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:07.668637 17740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:07.668668 17740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.668691 17740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:07.668716 17740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.673839 17740 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:45207
I20260928 17:49:07.673874 17868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:45207 every 8 connection(s)
I20260928 17:49:07.674129 17740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:49:07 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:07.680635 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 17740
I20260928 17:49:07.680740 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:49:07.680912 3960 external_mini_cluster.cc:1428] Setting key cbc15802a724306f9a388a5ba75d6751
I20260928 17:49:07.682413 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.674694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:56629 (local address 127.3.222.62:38581)
0928 17:49:07.674748 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:07.674750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:07.674926 (+ 176us) server_negotiation.cc:407] Connection header received
0928 17:49:07.675450 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:07.675452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:07.675462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:07.675521 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:07.676333 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.676832 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.677724 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.677920 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.677980 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.679860 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:07.679862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:07.679862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:07.679869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:07.681142 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.681481 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.681482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.681483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.681585 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.681837 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.681837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.681837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.681933 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:07.681975 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:07.682275 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 17:49:07.682350 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:49:07.682574 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.674664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:56629)
0928 17:49:07.674814 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:07.674851 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:07.675336 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:07.675578 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:07.675581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:07.676081 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:07.676264 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.676268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.676892 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.676892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.677639 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.677641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.677984 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.677984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.678069 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.678468 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:07.678475 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:07.679782 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:07.681213 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:07.681220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:07.681221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:07.681421 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:07.681652 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:07.681653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:07.681653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:07.681777 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:07.681981 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:07.681986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:07.682131 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 17:49:07.682233 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 17:49:07.682380 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260928 17:49:07.683015 17869 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38581
I20260928 17:49:07.683109 17869 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:07.683305 17869 heartbeater.cc:511] Master 127.3.222.62:38581 requested a full tablet report, sending...
I20260928 17:49:07.683765 17678 ts_manager.cc:194] Registered new tserver with Master: 59b3f406e90e41da95eeea9034e35f2e (127.3.222.1:45207)
I20260928 17:49:07.684572 17678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:56629
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:49:07.705864 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38581
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:07.782248 17874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:07.782421 17874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:07.782444 17874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:07.783561 17874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:07.783597 17874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:07.783622 17874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:49:07.785005 17874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:07.785203 17874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:07.785392 17874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:07.789966 17882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:07.790014 17880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:07.789965 17879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:07.790149 17874 server_base.cc:1034] running on GCE node
I20260928 17:49:07.790292 17874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:07.790473 17874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:07.791622 17874 hybrid_clock.cc:648] HybridClock initialized: now 1790617747791606 us; error 29 us; skew 500 ppm
Sep 28 17:49:07 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:07.793717 17874 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:49:07.794092 17874 webserver.cc:460] Webserver started at http://127.3.222.2:41347/ using document root <none> and password file <none>
I20260928 17:49:07.794265 17874 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:07.794302 17874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:07.794389 17874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:07.795296 17874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "316afb7358804495bcc19e323a81c98f"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "799b9b662c1bb2884210a68506416635"
server_key_iv: "5f211170f782ae5821809d8bc7318e1b"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.795568 17874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "316afb7358804495bcc19e323a81c98f"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "799b9b662c1bb2884210a68506416635"
server_key_iv: "5f211170f782ae5821809d8bc7318e1b"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.796622 17874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:49:07.797246 17889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.797385 17874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:07.797458 17874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "316afb7358804495bcc19e323a81c98f"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "799b9b662c1bb2884210a68506416635"
server_key_iv: "5f211170f782ae5821809d8bc7318e1b"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.797514 17874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:07.807927 17874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:07.809655 17874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:07.809772 17874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:07.809938 17874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:07.810181 17874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:07.810213 17874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.810240 17874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:07.810262 17874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.815451 17874 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:40479
I20260928 17:49:07.815508 18002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:40479 every 8 connection(s)
I20260928 17:49:07.815762 17874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:49:07.819486 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 17874
I20260928 17:49:07.819589 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:49:07.819765 3960 external_mini_cluster.cc:1428] Setting key 53b1b14c063198a2683a8caf2c6b4c1f
Sep 28 17:49:07 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:07.823992 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.816325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:60749 (local address 127.3.222.62:38581)
0928 17:49:07.816392 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:07.816394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:07.816560 (+ 166us) server_negotiation.cc:407] Connection header received
0928 17:49:07.816949 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:07.816950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:07.816960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:07.817011 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:07.817807 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.818289 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.819171 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.819372 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.819424 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.821649 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:07.821651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:07.821652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:07.821661 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:07.822782 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.823113 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.823114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.823115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.823167 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.823394 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.823395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.823396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.823503 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:07.823557 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:07.823823 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 17:49:07.823901 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:49:07.824231 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.816302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:60749)
0928 17:49:07.816431 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:07.816462 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:07.816873 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:07.817059 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:07.817062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:07.817553 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:07.817738 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.817743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.818347 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.818348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.819093 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.819095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.819443 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.819444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.819581 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.820034 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:07.820043 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:07.821546 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:07.822852 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:07.822858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:07.822859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:07.823050 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:07.823237 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:07.823238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:07.823238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:07.823340 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:07.823557 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:07.823562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:07.823688 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 17:49:07.823822 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 17:49:07.823984 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
I20260928 17:49:07.824604 18003 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38581
I20260928 17:49:07.824683 18003 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:07.824833 18003 heartbeater.cc:511] Master 127.3.222.62:38581 requested a full tablet report, sending...
I20260928 17:49:07.825150 17678 ts_manager.cc:194] Registered new tserver with Master: 316afb7358804495bcc19e323a81c98f (127.3.222.2:40479)
I20260928 17:49:07.825518 17678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:60749
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:49:07.844179 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38581
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:07.920332 18008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:07.920516 18008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:07.920537 18008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:07.921643 18008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:07.921679 18008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:07.921705 18008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:49:07.923061 18008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:07.923277 18008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:07.923480 18008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:07.928292 18013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:07.928287 18014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:07.928288 18016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:07.928797 18008 server_base.cc:1034] running on GCE node
I20260928 17:49:07.928999 18008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:07.929181 18008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:07.930316 18008 hybrid_clock.cc:648] HybridClock initialized: now 1790617747930303 us; error 28 us; skew 500 ppm
Sep 28 17:49:07 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:07.932601 18008 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:49:07.932973 18008 webserver.cc:460] Webserver started at http://127.3.222.3:38643/ using document root <none> and password file <none>
I20260928 17:49:07.933133 18008 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:07.933169 18008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:07.933254 18008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:07.934168 18008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "9be25b53ecd6462ba0421e66bb203ca3"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "c7fe643248ccb5540da5e6fecfceebd3"
server_key_iv: "2f95d15ad072f1f532d1c6752fbd4f4f"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.934433 18008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "9be25b53ecd6462ba0421e66bb203ca3"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "c7fe643248ccb5540da5e6fecfceebd3"
server_key_iv: "2f95d15ad072f1f532d1c6752fbd4f4f"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.935575 18008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:49:07.936206 18023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.936354 18008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:07.936429 18008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "9be25b53ecd6462ba0421e66bb203ca3"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "c7fe643248ccb5540da5e6fecfceebd3"
server_key_iv: "2f95d15ad072f1f532d1c6752fbd4f4f"
server_key_version: "encryptionkey@0"
I20260928 17:49:07.936479 18008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:07.957111 18008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:07.959080 18008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:07.959190 18008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:07.959383 18008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:07.959615 18008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:07.959643 18008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.959663 18008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:07.959681 18008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:07.964797 18008 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:40243
I20260928 17:49:07.964841 18136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:40243 every 8 connection(s)
I20260928 17:49:07.965121 18008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:49:07.968088 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 18008
I20260928 17:49:07.968168 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:49:07.968314 3960 external_mini_cluster.cc:1428] Setting key edd44e1862e69f7e278fccd4e5e4c1f9
Sep 28 17:49:07 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:07.973737 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.965752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:48435 (local address 127.3.222.62:38581)
0928 17:49:07.965813 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:07.965815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:07.966035 (+ 220us) server_negotiation.cc:407] Connection header received
0928 17:49:07.966489 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:07.966490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:07.966501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:07.966555 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:07.967505 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.968025 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.969278 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.969508 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.969563 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.971454 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:07.971456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:07.971457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:07.971463 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:07.972561 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.972900 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.972901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.972902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.972953 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.973164 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.973164 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.973165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.973266 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:07.973320 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:07.973572 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 17:49:07.973649 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:49:07.973881 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.965728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:48435)
0928 17:49:07.965879 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:07.965919 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:07.966371 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:07.966617 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:07.966627 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:07.967152 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:07.967382 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.967387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.968134 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.968135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.969209 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.969212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.969578 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.969579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.969706 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.970124 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:07.970134 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:07.971368 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:07.972633 (+ 1265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:07.972639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:07.972641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:07.972838 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:07.973014 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:07.973014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:07.973015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:07.973111 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:07.973319 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:07.973325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:07.973454 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 17:49:07.973542 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 17:49:07.973681 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260928 17:49:07.974267 18137 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38581
I20260928 17:49:07.974344 18137 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:07.974493 18137 heartbeater.cc:511] Master 127.3.222.62:38581 requested a full tablet report, sending...
I20260928 17:49:07.974808 17678 ts_manager.cc:194] Registered new tserver with Master: 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243)
I20260928 17:49:07.975162 17678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:48435
I20260928 17:49:07.979790 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:49:07.986342 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.981065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.3.222.62:38581)
0928 17:49:07.981102 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:07.981103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:07.981220 (+ 117us) server_negotiation.cc:407] Connection header received
0928 17:49:07.981277 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:07.981277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:07.981287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:07.981330 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:07.981831 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.982317 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.982998 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.983188 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.983275 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.984314 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:07.984316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:07.984316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:07.984322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:07.985333 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.985625 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.985626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.985626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.985694 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:07.985850 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:07.985850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:07.985851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:07.985980 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:07.986063 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:07.986187 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 17:49:07.986259 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:49:07.987612 17678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:49:07.989054 17678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:49:07.994433 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.990848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:45207 (local address 127.0.0.1:33586)
0928 17:49:07.991005 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:07.991051 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:07.991110 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:07.991511 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:07.991512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:07.991514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:07.991559 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.991560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.992455 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.992456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.993338 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.993341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.994154 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.994155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.994232 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.994246 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:49:07.994316 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 17:49:07.994366 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260928 17:49:07.994458 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.990958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.3.222.1:45207)
0928 17:49:07.991378 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:07.991381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:07.991390 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:49:07.991425 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:07.991427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:07.991440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:07.991503 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:07.991610 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.992366 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.993420 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.994074 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.994144 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.994200 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:07.994338 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 17:49:07.994375 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":77,"threads_started":1}
I20260928 17:49:07.994705 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.990865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:40243 (local address 127.0.0.1:55724)
0928 17:49:07.991178 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:07.991186 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:07.991250 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:07.991427 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:07.991429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:07.991431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:07.991476 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.991477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.992375 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.992376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.993611 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.993614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.994396 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.994397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.994451 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.994454 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:49:07.994602 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 17:49:07.994640 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":23,"thread_start_us":89,"threads_started":1}
I20260928 17:49:07.994990 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.991122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.3.222.3:40243)
0928 17:49:07.991284 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:07.991289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:07.991298 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:49:07.991331 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:07.991333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:07.991347 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:07.991418 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:07.991548 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.992290 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.993699 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.994318 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.994800 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.994861 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:07.994884 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 17:49:07.994922 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260928 17:49:07.995288 17803 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5fae72fcec4531ba29bad853b006d1 (DEFAULT_TABLE table=test-table [id=8d6b0c95a89844de8e7219e9308ec63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:07.995574 17803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5fae72fcec4531ba29bad853b006d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:07.995968 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.991048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40479 (local address 127.0.0.1:55040)
0928 17:49:07.992502 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:07.992512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:07.992567 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:07.992754 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:07.992755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:07.992757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:07.992789 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.992790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.993941 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.993941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.994952 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:07.994955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.995771 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.995772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:07.995828 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.995832 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:49:07.995870 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 17:49:07.995904 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"thread_start_us":69,"threads_started":1}
I20260928 17:49:07.996217 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:07.991180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.3.222.2:40479)
0928 17:49:07.991341 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:07.991345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:07.992586 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 17:49:07.992654 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:07.992656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:07.992672 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:07.992750 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:07.992847 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.993854 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.995040 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:07.995696 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:07.995757 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:07.996095 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:07.996119 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 17:49:07.996158 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260928 17:49:07.996830 17937 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5fae72fcec4531ba29bad853b006d1 (DEFAULT_TABLE table=test-table [id=8d6b0c95a89844de8e7219e9308ec63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:07.997159 17937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5fae72fcec4531ba29bad853b006d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:07.998771 18151 tablet_bootstrap.cc:492] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Bootstrap starting.
I20260928 17:49:07.998898 18071 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5fae72fcec4531ba29bad853b006d1 (DEFAULT_TABLE table=test-table [id=8d6b0c95a89844de8e7219e9308ec63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:07.999182 18071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5fae72fcec4531ba29bad853b006d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:07.999409 18151 tablet_bootstrap.cc:654] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:07.999877 18151 log.cc:824] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:08.000571 18151 tablet_bootstrap.cc:492] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: No bootstrap required, opened a new log
I20260928 17:49:08.000634 18151 ts_tablet_manager.cc:1397] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:49:08.001842 18154 tablet_bootstrap.cc:492] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: Bootstrap starting.
I20260928 17:49:08.001924 18153 tablet_bootstrap.cc:492] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Bootstrap starting.
I20260928 17:49:08.002216 18151 raft_consensus.cc:348] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.002380 18151 raft_consensus.cc:374] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:08.002420 18151 raft_consensus.cc:729] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3f406e90e41da95eeea9034e35f2e, State: Initialized, Role: FOLLOWER
I20260928 17:49:08.002472 18153 tablet_bootstrap.cc:654] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:08.002472 18154 tablet_bootstrap.cc:654] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:08.002547 18151 consensus_queue.cc:260] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.002782 18151 ts_tablet_manager.cc:1428] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:49:08.002898 17869 heartbeater.cc:503] Master 127.3.222.62:38581 was elected leader, sending a full tablet report...
I20260928 17:49:08.002995 18153 log.cc:824] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:08.002990 18154 log.cc:824] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:08.003680 18153 tablet_bootstrap.cc:492] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: No bootstrap required, opened a new log
I20260928 17:49:08.003692 18154 tablet_bootstrap.cc:492] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: No bootstrap required, opened a new log
I20260928 17:49:08.003746 18153 ts_tablet_manager.cc:1397] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:49:08.003746 18154 ts_tablet_manager.cc:1397] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:49:08.005522 18153 raft_consensus.cc:348] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.005522 18154 raft_consensus.cc:348] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.005692 18154 raft_consensus.cc:374] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:08.005692 18153 raft_consensus.cc:374] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:08.005726 18154 raft_consensus.cc:729] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be25b53ecd6462ba0421e66bb203ca3, State: Initialized, Role: FOLLOWER
I20260928 17:49:08.005726 18153 raft_consensus.cc:729] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316afb7358804495bcc19e323a81c98f, State: Initialized, Role: FOLLOWER
I20260928 17:49:08.005839 18154 consensus_queue.cc:260] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.005839 18153 consensus_queue.cc:260] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.006088 18154 ts_tablet_manager.cc:1428] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:49:08.006088 18153 ts_tablet_manager.cc:1428] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:49:08.006134 18003 heartbeater.cc:503] Master 127.3.222.62:38581 was elected leader, sending a full tablet report...
I20260928 17:49:08.006202 18137 heartbeater.cc:503] Master 127.3.222.62:38581 was elected leader, sending a full tablet report...
W20260928 17:49:08.066259 18004 tablet.cc:2369] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:49:08.077200 18158 raft_consensus.cc:482] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:49:08.077332 18158 raft_consensus.cc:504] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.077729 18158 leader_election.cc:288] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59b3f406e90e41da95eeea9034e35f2e (127.3.222.1:45207), 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243)
I20260928 17:49:08.080494 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.077974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:55633 (local address 127.3.222.1:45207)
0928 17:49:08.078038 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.078040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.078216 (+ 176us) server_negotiation.cc:407] Connection header received
0928 17:49:08.078247 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.078248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.078260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.078318 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:08.078456 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.078898 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.079805 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.080193 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.080250 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.080261 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.080399 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.080431 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:49:08.080503 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.078019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:57691 (local address 127.3.222.3:40243)
0928 17:49:08.078069 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.078070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.078078 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:49:08.078104 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.078104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.078113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.078164 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:08.078334 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.078855 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.079812 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.080192 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.080252 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.080262 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.080397 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.080438 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:49:08.080523 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.077900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:40243 (local address 127.3.222.2:57691)
0928 17:49:08.077972 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.077984 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.078034 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.078179 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.078180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.078182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:08.078250 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.078252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.078921 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.078922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.079727 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.079730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.080262 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.080263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.080331 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.080334 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.080377 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.080414 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":4}
I20260928 17:49:08.080580 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.077903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:45207 (local address 127.3.222.2:55633)
0928 17:49:08.078165 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.078173 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.078214 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.078332 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.078333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.078334 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:08.078399 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.078401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.078969 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.078969 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.079731 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.079733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.080262 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.080263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.080337 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.080340 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.080388 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.080461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":112,"thread_start_us":98,"threads_started":1}
I20260928 17:49:08.080873 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5fae72fcec4531ba29bad853b006d1" candidate_uuid: "316afb7358804495bcc19e323a81c98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3f406e90e41da95eeea9034e35f2e" is_pre_election: true
I20260928 17:49:08.080856 18091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5fae72fcec4531ba29bad853b006d1" candidate_uuid: "316afb7358804495bcc19e323a81c98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be25b53ecd6462ba0421e66bb203ca3" is_pre_election: true
I20260928 17:49:08.081025 17823 raft_consensus.cc:2454] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316afb7358804495bcc19e323a81c98f in term 0.
I20260928 17:49:08.081017 18091 raft_consensus.cc:2454] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316afb7358804495bcc19e323a81c98f in term 0.
I20260928 17:49:08.081238 17890 leader_election.cc:302] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316afb7358804495bcc19e323a81c98f, 59b3f406e90e41da95eeea9034e35f2e; no voters:
I20260928 17:49:08.081362 18158 raft_consensus.cc:2781] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:49:08.081418 18158 raft_consensus.cc:482] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:49:08.081439 18158 raft_consensus.cc:3037] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:08.082197 18158 raft_consensus.cc:504] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.082305 18158 leader_election.cc:288] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [CANDIDATE]: Term 1 election: Requested vote from peers 59b3f406e90e41da95eeea9034e35f2e (127.3.222.1:45207), 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243)
I20260928 17:49:08.082445 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5fae72fcec4531ba29bad853b006d1" candidate_uuid: "316afb7358804495bcc19e323a81c98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3f406e90e41da95eeea9034e35f2e"
I20260928 17:49:08.082504 18091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5fae72fcec4531ba29bad853b006d1" candidate_uuid: "316afb7358804495bcc19e323a81c98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be25b53ecd6462ba0421e66bb203ca3"
I20260928 17:49:08.082520 17823 raft_consensus.cc:3037] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:08.082556 18091 raft_consensus.cc:3037] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:08.083181 18091 raft_consensus.cc:2454] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316afb7358804495bcc19e323a81c98f in term 1.
I20260928 17:49:08.083232 17823 raft_consensus.cc:2454] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316afb7358804495bcc19e323a81c98f in term 1.
I20260928 17:49:08.083359 17891 leader_election.cc:302] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316afb7358804495bcc19e323a81c98f, 9be25b53ecd6462ba0421e66bb203ca3; no voters:
I20260928 17:49:08.083458 18158 raft_consensus.cc:2781] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:49:08.083583 18158 raft_consensus.cc:686] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 1 LEADER]: Becoming Leader. State: Replica: 316afb7358804495bcc19e323a81c98f, State: Running, Role: LEADER
I20260928 17:49:08.083657 18158 consensus_queue.cc:237] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.084386 17678 catalog_manager.cc:5495] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f reported cstate change: term changed from 0 to 1, leader changed from <none> to 316afb7358804495bcc19e323a81c98f (127.3.222.2). New cstate: current_term: 1 leader_uuid: "316afb7358804495bcc19e323a81c98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } health_report { overall_health: HEALTHY } } }
I20260928 17:49:08.109153 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.106942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.3.222.2:40479)
0928 17:49:08.107025 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.107026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.107035 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:49:08.107066 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.107067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.107085 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.107148 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:49:08.107277 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.107747 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.108434 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.108608 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.108708 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.108852 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:49:08.108946 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:49:08.109009 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.109023 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.109040 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 17:49:08.110853 17823 raft_consensus.cc:1261] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Refusing update from remote peer 316afb7358804495bcc19e323a81c98f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:08.110857 18091 raft_consensus.cc:1261] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [term 1 FOLLOWER]: Refusing update from remote peer 316afb7358804495bcc19e323a81c98f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:08.111140 18158 consensus_queue.cc:1035] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:08.111294 18161 consensus_queue.cc:1035] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:08.114319 18165 mvcc.cc:205] Tried to move back new op lower bound from 7334370296259620864 to 7334370296151166976. Current Snapshot: MvccSnapshot[applied={T|T < 7334370296259620864}]
I20260928 17:49:08.114420 18166 mvcc.cc:205] Tried to move back new op lower bound from 7334370296259620864 to 7334370296151166976. Current Snapshot: MvccSnapshot[applied={T|T < 7334370296259620864 or (T in {7334370296259620864})}]
I20260928 17:49:08.115582 17677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:08.115700 17677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:08.116457 17677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Sending DeleteTablet for 3 replicas of tablet 1f5fae72fcec4531ba29bad853b006d1
I20260928 17:49:08.116763 18071 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5fae72fcec4531ba29bad853b006d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:08 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:55724
I20260928 17:49:08.116796 17937 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5fae72fcec4531ba29bad853b006d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:08 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:55040
I20260928 17:49:08.116842 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 17740
I20260928 17:49:08.116961 17803 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5fae72fcec4531ba29bad853b006d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:08 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:33586
I20260928 17:49:08.119318 18174 tablet_replica.cc:330] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: stopping tablet replica
I20260928 17:49:08.119421 18174 raft_consensus.cc:2229] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:49:08.119520 18174 raft_consensus.cc:2258] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:08.119714 18175 tablet_replica.cc:330] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: stopping tablet replica
I20260928 17:49:08.119803 18175 raft_consensus.cc:2229] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:49:08.119887 18175 raft_consensus.cc:2258] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:08.119959 18174 ts_tablet_manager.cc:1905] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:08.120256 18175 ts_tablet_manager.cc:1905] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:08.121538 18174 ts_tablet_manager.cc:1918] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 17:49:08.121790 17662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:45207 (error 108)
I20260928 17:49:08.121940 18174 log.cc:1196] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/1f5fae72fcec4531ba29bad853b006d1
I20260928 17:49:08.121934 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 17874
I20260928 17:49:08.122733 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.122608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:45207 (local address 127.0.0.1:33602)
0928 17:49:08.122665 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.122701 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:45207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 17:49:08.122753 18174 ts_tablet_manager.cc:1939] T 1f5fae72fcec4531ba29bad853b006d1 P 9be25b53ecd6462ba0421e66bb203ca3: Deleting consensus metadata
W20260928 17:49:08.122844 17662 catalog_manager.cc:4584] TS 59b3f406e90e41da95eeea9034e35f2e (127.3.222.1:45207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f5fae72fcec4531ba29bad853b006d1: Network error: Client connection negotiation failed: client connection to 127.3.222.1:45207: connect: Connection refused (error 111)
I20260928 17:49:08.123121 17663 catalog_manager.cc:4841] TS 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243): tablet 1f5fae72fcec4531ba29bad853b006d1 (table test-table [id=8d6b0c95a89844de8e7219e9308ec63b]) successfully deleted
I20260928 17:49:08.126811 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 18008
I20260928 17:49:08.126966 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.126861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40479 (local address 127.0.0.1:55058)
0928 17:49:08.126909 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.126939 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.2:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 17:49:08.130826 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 17646
I20260928 17:49:08.135154 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38581
--webserver_interface=127.3.222.62
--webserver_port=34285
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:38581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:08.213825 18177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:08.214004 18177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:08.214025 18177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:08.215126 18177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:49:08.215164 18177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:08.215178 18177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:08.215188 18177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:49:08.215202 18177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:49:08.216650 18177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:38581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=34285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:08.216864 18177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:08.217054 18177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:08.224994 18186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:08.225068 18184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.225078 18177 server_base.cc:1034] running on GCE node
W20260928 17:49:08.225024 18183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.225368 18177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:08.225613 18177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:08.226783 18177 hybrid_clock.cc:648] HybridClock initialized: now 1790617748226770 us; error 29 us; skew 500 ppm
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:08.229187 18177 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:49:08.229562 18177 webserver.cc:460] Webserver started at http://127.3.222.62:34285/ using document root <none> and password file <none>
I20260928 17:49:08.229751 18177 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:08.229799 18177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:08.231036 18177 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:49:08.231659 18193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.231799 18177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.231876 18177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "c900a3e55b2447db96c4721ca6f6d142"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "9f4bdfb6a37cc2613a92f1458f9ee9f3"
server_key_iv: "d82e5ef7bb2d2e70ae432950f0b9a7a2"
server_key_version: "encryptionkey@0"
I20260928 17:49:08.232249 18177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:08.250566 18177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:08.252580 18177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:08.252698 18177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:08.256268 18177 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:38581
I20260928 17:49:08.256664 18177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:49:08.258847 18246 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:49:08.259322 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 18177
I20260928 17:49:08.259334 18245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:38581 every 8 connection(s)
I20260928 17:49:08.259647 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:45207
--local_ip_for_outbound_sockets=127.3.222.1
--tserver_master_addrs=127.3.222.62:38581
--webserver_port=36859
--webserver_interface=127.3.222.1
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:08.259745 18246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Bootstrap starting.
I20260928 17:49:08.268200 18246 log.cc:824] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:08.269989 18246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:49:08.270244 18246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Bootstrap complete.
I20260928 17:49:08.272308 18246 raft_consensus.cc:348] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:08.272548 18246 raft_consensus.cc:729] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c900a3e55b2447db96c4721ca6f6d142, State: Initialized, Role: FOLLOWER
I20260928 17:49:08.272703 18246 consensus_queue.cc:260] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:08.272775 18246 raft_consensus.cc:388] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:49:08.272827 18246 raft_consensus.cc:482] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:49:08.272872 18246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:49:08.273801 18246 raft_consensus.cc:504] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:08.273890 18246 leader_election.cc:302] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c900a3e55b2447db96c4721ca6f6d142; no voters:
I20260928 17:49:08.274044 18246 leader_election.cc:288] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 17:49:08.274087 18250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:49:08.274204 18250 raft_consensus.cc:686] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 2 LEADER]: Becoming Leader. State: Replica: c900a3e55b2447db96c4721ca6f6d142, State: Running, Role: LEADER
I20260928 17:49:08.274284 18246 sys_catalog.cc:564] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:49:08.274282 18250 consensus_queue.cc:237] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:08.274608 18250 sys_catalog.cc:455] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c900a3e55b2447db96c4721ca6f6d142. Latest consensus state: current_term: 2 leader_uuid: "c900a3e55b2447db96c4721ca6f6d142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } } }
I20260928 17:49:08.274677 18250 sys_catalog.cc:458] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:08.274814 18252 sys_catalog.cc:455] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c900a3e55b2447db96c4721ca6f6d142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } } }
I20260928 17:49:08.274871 18252 sys_catalog.cc:458] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:08.275121 18254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:49:08.275899 18254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:49:08.276002 18254 catalog_manager.cc:1259] Loaded cluster ID: 50eaabda8e124023b332028b8a5fa835
I20260928 17:49:08.276044 18254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:49:08.276265 18254 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 17:49:08.278954 18254 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 17:49:08.284119 18254 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:49:08.284299 18254 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Loaded TSK: 0
I20260928 17:49:08.284484 18254 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 17:49:08.353036 18248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:08.353247 18248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:08.353276 18248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:08.354398 18248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:08.354434 18248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:08.354468 18248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:49:08.355916 18248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:45207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=36859
--tserver_master_addrs=127.3.222.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:08.356140 18248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:08.356331 18248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:08.361860 18276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.361963 18248 server_base.cc:1034] running on GCE node
W20260928 17:49:08.361974 18274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:08.362151 18273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.362362 18248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:08.362635 18248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:08.363812 18248 hybrid_clock.cc:648] HybridClock initialized: now 1790617748363784 us; error 38 us; skew 500 ppm
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:08.366629 18248 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:49:08.367136 18248 webserver.cc:460] Webserver started at http://127.3.222.1:36859/ using document root <none> and password file <none>
I20260928 17:49:08.367419 18248 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:08.367512 18248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:08.369168 18248 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:49:08.369874 18283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.370029 18248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:08.370121 18248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "59b3f406e90e41da95eeea9034e35f2e"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "e1eb72288d0e1a45b012a0718d774d7b"
server_key_iv: "beb253e73d407a121291e39cb67d29bf"
server_key_version: "encryptionkey@0"
I20260928 17:49:08.370540 18248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:08.409783 18248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:08.412190 18248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:08.412336 18248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:08.412652 18248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:08.413168 18290 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:49:08.414549 18248 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:49:08.414605 18248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:49:08.414635 18248 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:49:08.415437 18248 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:49:08.415475 18248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260928 17:49:08.415544 18290 tablet_bootstrap.cc:492] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Bootstrap starting.
I20260928 17:49:08.421825 18248 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:45207
I20260928 17:49:08.422083 18397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:45207 every 8 connection(s)
I20260928 17:49:08.422252 18248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:49:08.425299 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 18248
I20260928 17:49:08.425459 18290 log.cc:824] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:08.425585 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:40479
--local_ip_for_outbound_sockets=127.3.222.2
--tserver_master_addrs=127.3.222.62:38581
--webserver_port=41347
--webserver_interface=127.3.222.2
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:08.429052 18290 tablet_bootstrap.cc:492] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:49:08.429344 18290 tablet_bootstrap.cc:492] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Bootstrap complete.
I20260928 17:49:08.430333 18290 ts_tablet_manager.cc:1397] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:08.431501 18290 raft_consensus.cc:348] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.431778 18290 raft_consensus.cc:729] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3f406e90e41da95eeea9034e35f2e, State: Initialized, Role: FOLLOWER
I20260928 17:49:08.431897 18290 consensus_queue.cc:260] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } }
I20260928 17:49:08.432118 18290 ts_tablet_manager.cc:1428] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
I20260928 17:49:08.434954 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.422927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:58441)
0928 17:49:08.423103 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.423149 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.423839 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.424345 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.424350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.424759 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:08.425198 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.425205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.426655 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.426656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.427858 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.427861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.428679 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.428680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.428814 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.429448 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:08.429458 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:08.431289 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:08.433418 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:08.433424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:08.433426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:08.433691 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:08.433958 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:08.433958 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:08.433959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:08.434066 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:08.434480 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:08.434486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:08.434600 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.434675 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.434788 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260928 17:49:08.435083 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.423039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:58441 (local address 127.3.222.62:38581)
0928 17:49:08.423339 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.423349 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.423396 (+ 47us) server_negotiation.cc:407] Connection header received
0928 17:49:08.424073 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.424082 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.424137 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.424277 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:08.425344 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.426538 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.427929 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.428476 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.428574 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.431410 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:08.431415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:08.431415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:08.431428 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:08.433336 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.433784 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.433787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.433788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.433899 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.434137 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.434138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.434138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.434419 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:08.434502 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.434767 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.434898 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":80,"threads_started":1}
I20260928 17:49:08.435689 18398 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38581
I20260928 17:49:08.435784 18398 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:08.435956 18398 heartbeater.cc:511] Master 127.3.222.62:38581 requested a full tablet report, sending...
I20260928 17:49:08.436658 18210 ts_manager.cc:194] Registered new tserver with Master: 59b3f406e90e41da95eeea9034e35f2e (127.3.222.1:45207)
I20260928 17:49:08.437594 18210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:58441
I20260928 17:49:08.438432 18398 heartbeater.cc:503] Master 127.3.222.62:38581 was elected leader, sending a full tablet report...
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:49:08.444634 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.437165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:45207 (local address 127.0.0.1:33608)
0928 17:49:08.437312 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.437353 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.437412 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.437989 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.437995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.438223 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:08.438262 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.438263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.439192 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.439193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.439773 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.439775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.440188 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.440190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.440262 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.440641 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:08.440645 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:08.441971 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:08.443667 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:08.443673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:08.443674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:08.443841 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:08.444023 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:08.444024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:08.444024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:08.444074 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:08.444353 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:08.444359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:08.444403 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.444479 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.444562 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 17:49:08.444692 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.437218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.3.222.1:45207)
0928 17:49:08.437855 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.437860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.437870 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:49:08.437898 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.437900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.437915 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.437973 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:08.438331 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.439128 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.439844 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.440127 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.440181 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.442049 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:08.442057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:08.442057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:08.442064 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:08.443583 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.443907 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.443910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.443910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.443965 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.444125 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.444125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.444126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.444272 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:08.444343 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.444518 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.444603 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":69,"threads_started":1}
I20260928 17:49:08.444964 18332 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5fae72fcec4531ba29bad853b006d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:08 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:33608
I20260928 17:49:08.445473 18407 tablet_replica.cc:330] stopping tablet replica
I20260928 17:49:08.445604 18407 raft_consensus.cc:2229] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:49:08.445669 18407 raft_consensus.cc:2258] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:08.445917 18407 ts_tablet_manager.cc:1905] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:08.447925 18407 ts_tablet_manager.cc:1918] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:49:08.448000 18407 log.cc:1196] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/1f5fae72fcec4531ba29bad853b006d1
I20260928 17:49:08.448170 18407 ts_tablet_manager.cc:1939] T 1f5fae72fcec4531ba29bad853b006d1 P 59b3f406e90e41da95eeea9034e35f2e: Deleting consensus metadata
I20260928 17:49:08.448593 18194 catalog_manager.cc:4841] TS 59b3f406e90e41da95eeea9034e35f2e (127.3.222.1:45207): tablet 1f5fae72fcec4531ba29bad853b006d1 (table test-table [id=8d6b0c95a89844de8e7219e9308ec63b]) successfully deleted
W20260928 17:49:08.508136 18403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:08.508311 18403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:08.508332 18403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:08.509433 18403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:08.509469 18403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:08.509492 18403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:49:08.510896 18403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:40479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=41347
--tserver_master_addrs=127.3.222.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:08.511089 18403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:08.511300 18403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:49:08.516019 18403 server_base.cc:1034] running on GCE node
W20260928 17:49:08.515936 18412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:08.515951 18413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:08.515936 18415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.516309 18403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:08.516517 18403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:08.517659 18403 hybrid_clock.cc:648] HybridClock initialized: now 1790617748517643 us; error 30 us; skew 500 ppm
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:08.527047 18403 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:49:08.527472 18403 webserver.cc:460] Webserver started at http://127.3.222.2:41347/ using document root <none> and password file <none>
I20260928 17:49:08.527664 18403 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:08.527714 18403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:08.528914 18403 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:49:08.529512 18422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.529670 18403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:08.529742 18403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "316afb7358804495bcc19e323a81c98f"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "799b9b662c1bb2884210a68506416635"
server_key_iv: "5f211170f782ae5821809d8bc7318e1b"
server_key_version: "encryptionkey@0"
I20260928 17:49:08.530050 18403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:08.553925 18403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:08.555704 18403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:08.555819 18403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:08.556033 18403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:08.556455 18429 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 17:49:08.557766 18429 ts_tablet_manager.cc:1852] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 17:49:08.558013 18429 ts_tablet_manager.cc:1905] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:08.560128 18429 ts_tablet_manager.cc:1918] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:49:08.560207 18429 log.cc:1196] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/1f5fae72fcec4531ba29bad853b006d1
I20260928 17:49:08.560390 18429 ts_tablet_manager.cc:1939] T 1f5fae72fcec4531ba29bad853b006d1 P 316afb7358804495bcc19e323a81c98f: Deleting consensus metadata
I20260928 17:49:08.560586 18403 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 17:49:08.560640 18403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 17:49:08.560664 18403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:08.560678 18403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.566061 18403 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:40479
I20260928 17:49:08.566129 18536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:40479 every 8 connection(s)
I20260928 17:49:08.566370 18403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:49:08.571342 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 18403
I20260928 17:49:08.571596 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:40243
--local_ip_for_outbound_sockets=127.3.222.3
--tserver_master_addrs=127.3.222.62:38581
--webserver_port=38643
--webserver_interface=127.3.222.3
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:08.574692 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.567004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:48645 (local address 127.3.222.62:38581)
0928 17:49:08.567067 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.567069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.567262 (+ 193us) server_negotiation.cc:407] Connection header received
0928 17:49:08.567738 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.567739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.567748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.567797 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:08.568383 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.568871 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.569818 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.570005 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.570049 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.572135 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:08.572136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:08.572137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:08.572144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:08.573360 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.573770 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.573772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.573772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.573829 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.574059 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.574060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.574060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.574185 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:08.574253 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.574513 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.574599 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:49:08.574759 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.566972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:48645)
0928 17:49:08.567137 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.567173 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.567657 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.567863 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.567867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.568124 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:08.568305 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.568309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.568933 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.568934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.569755 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.569757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.570058 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.570059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.570151 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.570552 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:08.570559 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:08.572045 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:08.573440 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:08.573447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:08.573449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:08.573702 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:08.573886 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:08.573886 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:08.573887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:08.574003 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:08.574240 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:08.574244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:08.574348 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.574425 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.574571 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260928 17:49:08.575336 18537 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38581
I20260928 17:49:08.575418 18537 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:08.575593 18537 heartbeater.cc:511] Master 127.3.222.62:38581 requested a full tablet report, sending...
I20260928 17:49:08.575944 18210 ts_manager.cc:194] Registered new tserver with Master: 316afb7358804495bcc19e323a81c98f (127.3.222.2:40479)
I20260928 17:49:08.576319 18210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:48645
W20260928 17:49:08.649231 18540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:08.649413 18540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:08.649443 18540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:08.650520 18540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:08.650556 18540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:08.650589 18540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:49:08.652024 18540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:40243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=38643
--tserver_master_addrs=127.3.222.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:08.652232 18540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:08.652418 18540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:08.657027 18546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:08.657032 18548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.657029 18540 server_base.cc:1034] running on GCE node
W20260928 17:49:08.657127 18545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.657349 18540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:08.657553 18540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:08.658684 18540 hybrid_clock.cc:648] HybridClock initialized: now 1790617748658673 us; error 28 us; skew 500 ppm
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:08.660872 18540 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:49:08.661268 18540 webserver.cc:460] Webserver started at http://127.3.222.3:38643/ using document root <none> and password file <none>
I20260928 17:49:08.661450 18540 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:08.661505 18540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:08.662734 18540 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:49:08.663302 18555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.663435 18540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.663511 18540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "9be25b53ecd6462ba0421e66bb203ca3"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "c7fe643248ccb5540da5e6fecfceebd3"
server_key_iv: "2f95d15ad072f1f532d1c6752fbd4f4f"
server_key_version: "encryptionkey@0"
I20260928 17:49:08.663816 18540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:08.673138 18540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:08.674909 18540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:08.675024 18540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:08.675243 18540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:08.675525 18540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:08.675558 18540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.675583 18540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:08.675607 18540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.680845 18540 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:40243
I20260928 17:49:08.680902 18668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:40243 every 8 connection(s)
I20260928 17:49:08.681198 18540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:49:08.686419 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 18540
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:08.689706 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.681844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:48041 (local address 127.3.222.62:38581)
0928 17:49:08.681923 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.681925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.682115 (+ 190us) server_negotiation.cc:407] Connection header received
0928 17:49:08.682566 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.682567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.682580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.682630 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:08.683490 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.683956 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.684894 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.685114 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.685177 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.687073 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:08.687074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:08.687074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:08.687080 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:08.688486 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.688826 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.688827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.688827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.688885 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.689134 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.689134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.689135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.689235 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:08.689293 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.689532 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.689618 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 17:49:08.689831 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.681791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:48041)
0928 17:49:08.681957 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.681998 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.682490 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.682693 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.682696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.683225 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:08.683421 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.683426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.684016 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.684017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.684786 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.684788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.685187 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.685188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.685286 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.685704 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:08.685711 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:08.687003 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:08.688559 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:08.688564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:08.688566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:08.688766 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:08.688937 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:08.688938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:08.688938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:08.689053 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:08.689296 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:08.689301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:08.689410 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.689493 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.689632 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260928 17:49:08.690205 18669 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38581
I20260928 17:49:08.690299 18669 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:08.690464 18669 heartbeater.cc:511] Master 127.3.222.62:38581 requested a full tablet report, sending...
I20260928 17:49:08.690845 18210 ts_manager.cc:194] Registered new tserver with Master: 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243)
I20260928 17:49:08.691295 18210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:48041
I20260928 17:49:08.692698 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.686746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.3.222.62:38581)
0928 17:49:08.686869 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.686871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.686879 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:49:08.686900 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.686900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.686910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.686962 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:08.687607 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.688058 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.688925 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.689195 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.689241 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.690460 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:08.690461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:08.690461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:08.690468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:08.691908 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.692185 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.692186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.692186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.692234 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:08.692350 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:08.692350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:08.692350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:08.692431 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:08.692548 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.692581 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.692638 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260928 17:49:08.693241 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:49:08.695482 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.693521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.3.222.62:38581)
0928 17:49:08.693574 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.693576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.693633 (+ 57us) server_negotiation.cc:407] Connection header received
0928 17:49:08.693686 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.693686 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.693710 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.693766 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:49:08.693829 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.694251 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.694897 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.695069 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.695112 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.695203 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:49:08.695327 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:49:08.695381 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.695394 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.695411 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:49:08.695699 18210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:49:08.697162 18210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:49:08.699170 18332 tablet_service.cc:1467] Processing CreateTablet for tablet cc7e327fb27942548b10cb0b5f02fa5f (DEFAULT_TABLE table=test-table [id=00836448daed40a0b6e95cd1841435b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:08.699364 18332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7e327fb27942548b10cb0b5f02fa5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:08.700582 18290 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Bootstrap starting.
I20260928 17:49:08.700871 18290 tablet_bootstrap.cc:654] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:08.701638 18290 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: No bootstrap required, opened a new log
I20260928 17:49:08.701681 18290 ts_tablet_manager.cc:1397] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:49:08.701817 18290 raft_consensus.cc:348] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:08.701884 18290 raft_consensus.cc:374] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:08.701928 18290 raft_consensus.cc:729] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3f406e90e41da95eeea9034e35f2e, State: Initialized, Role: FOLLOWER
I20260928 17:49:08.701972 18290 consensus_queue.cc:260] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:08.702066 18290 ts_tablet_manager.cc:1428] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.702356 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.698801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40479 (local address 127.0.0.1:55074)
0928 17:49:08.698830 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.698841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.698876 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.699243 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.699244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.699246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:08.699292 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.699293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.700245 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.700246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.701232 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.701234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.702127 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.702127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.702202 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.702222 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.702250 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.702290 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":2}
I20260928 17:49:08.702723 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.698908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.3.222.2:40479)
0928 17:49:08.699078 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.699083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.699092 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:49:08.699127 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.699129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.699143 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.699232 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:08.699344 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.700166 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.701320 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.702046 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.702509 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.702575 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.702600 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.702646 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260928 17:49:08.703152 18471 tablet_service.cc:1467] Processing CreateTablet for tablet cc7e327fb27942548b10cb0b5f02fa5f (DEFAULT_TABLE table=test-table [id=00836448daed40a0b6e95cd1841435b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:08.703480 18471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7e327fb27942548b10cb0b5f02fa5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:08.704219 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.698886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:40243 (local address 127.0.0.1:55736)
0928 17:49:08.699005 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.699014 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.699050 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.700912 (+ 1862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.700913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.700915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:08.700954 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.700955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.702116 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.702127 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.703088 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.703090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.703984 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.703985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.704053 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.704056 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.704102 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.704152 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260928 17:49:08.704442 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.700594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.3.222.3:40243)
0928 17:49:08.700780 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.700784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.700792 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:49:08.700823 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.700825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.700836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.700962 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:08.701020 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.702016 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.703168 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.703893 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.703971 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.704040 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.704350 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.704388 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260928 17:49:08.705035 18603 tablet_service.cc:1467] Processing CreateTablet for tablet cc7e327fb27942548b10cb0b5f02fa5f (DEFAULT_TABLE table=test-table [id=00836448daed40a0b6e95cd1841435b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:08.705339 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7e327fb27942548b10cb0b5f02fa5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:08.705567 18429 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Bootstrap starting.
I20260928 17:49:08.706297 18429 tablet_bootstrap.cc:654] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:08.706593 18429 log.cc:824] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:08.707355 18429 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: No bootstrap required, opened a new log
I20260928 17:49:08.707410 18429 ts_tablet_manager.cc:1397] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:49:08.708564 18680 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Bootstrap starting.
I20260928 17:49:08.709093 18680 tablet_bootstrap.cc:654] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:08.709106 18429 raft_consensus.cc:348] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:08.709251 18429 raft_consensus.cc:374] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:08.709292 18429 raft_consensus.cc:729] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316afb7358804495bcc19e323a81c98f, State: Initialized, Role: FOLLOWER
I20260928 17:49:08.709401 18429 consensus_queue.cc:260] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:08.709529 18680 log.cc:824] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:08.709656 18537 heartbeater.cc:503] Master 127.3.222.62:38581 was elected leader, sending a full tablet report...
I20260928 17:49:08.709641 18429 ts_tablet_manager.cc:1428] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:49:08.710258 18680 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: No bootstrap required, opened a new log
I20260928 17:49:08.710322 18680 ts_tablet_manager.cc:1397] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:49:08.711674 18680 raft_consensus.cc:348] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:08.711793 18680 raft_consensus.cc:374] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:08.711812 18680 raft_consensus.cc:729] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be25b53ecd6462ba0421e66bb203ca3, State: Initialized, Role: FOLLOWER
I20260928 17:49:08.711889 18680 consensus_queue.cc:260] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:08.712095 18680 ts_tablet_manager.cc:1428] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:49:08.712131 18669 heartbeater.cc:503] Master 127.3.222.62:38581 was elected leader, sending a full tablet report...
I20260928 17:49:08.713811 18404 raft_consensus.cc:482] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:49:08.713888 18404 raft_consensus.cc:504] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:08.714154 18404 leader_election.cc:288] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316afb7358804495bcc19e323a81c98f (127.3.222.2:40479), 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243)
I20260928 17:49:08.716722 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.714409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:42391 (local address 127.3.222.3:40243)
0928 17:49:08.714441 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.714442 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.714555 (+ 113us) server_negotiation.cc:407] Connection header received
0928 17:49:08.714608 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.714608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.714617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.714658 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:08.714785 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.715191 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.716082 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.716427 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.716483 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.716490 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.716641 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.716670 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:49:08.716727 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.714338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:53017 (local address 127.3.222.2:40479)
0928 17:49:08.714373 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.714375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.714383 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:49:08.714469 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.714470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.714480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.714549 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:08.714707 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.715170 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.716071 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.716425 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.716478 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.716487 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.716637 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.716668 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:49:08.716737 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.714278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40479 (local address 127.3.222.1:53017)
0928 17:49:08.714337 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.714347 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.714391 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.714580 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.714581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.714583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:08.714653 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.714655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.715266 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.715267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.715994 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.715996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.716490 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.716491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.716563 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.716580 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.716626 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.716663 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":2}
I20260928 17:49:08.716950 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.714373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:40243 (local address 127.3.222.1:42391)
0928 17:49:08.714504 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.714513 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:08.714564 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:08.714660 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:08.714661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:08.714662 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:08.714733 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.714734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.715270 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.715271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.716008 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:08.716010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.716507 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.716508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:08.716562 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.716577 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 17:49:08.716631 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:49:08.716667 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":11,"thread_start_us":84,"threads_started":1}
I20260928 17:49:08.717018 18491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7e327fb27942548b10cb0b5f02fa5f" candidate_uuid: "59b3f406e90e41da95eeea9034e35f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316afb7358804495bcc19e323a81c98f" is_pre_election: true
I20260928 17:49:08.717140 18491 raft_consensus.cc:2454] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b3f406e90e41da95eeea9034e35f2e in term 0.
I20260928 17:49:08.717206 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7e327fb27942548b10cb0b5f02fa5f" candidate_uuid: "59b3f406e90e41da95eeea9034e35f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be25b53ecd6462ba0421e66bb203ca3" is_pre_election: true
I20260928 17:49:08.717325 18623 raft_consensus.cc:2454] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b3f406e90e41da95eeea9034e35f2e in term 0.
I20260928 17:49:08.717326 18285 leader_election.cc:302] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316afb7358804495bcc19e323a81c98f, 59b3f406e90e41da95eeea9034e35f2e; no voters:
I20260928 17:49:08.717422 18404 raft_consensus.cc:2781] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:49:08.717485 18404 raft_consensus.cc:482] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:49:08.717507 18404 raft_consensus.cc:3037] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:08.718075 18404 raft_consensus.cc:504] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:08.718173 18404 leader_election.cc:288] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 1 election: Requested vote from peers 316afb7358804495bcc19e323a81c98f (127.3.222.2:40479), 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243)
I20260928 17:49:08.718327 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7e327fb27942548b10cb0b5f02fa5f" candidate_uuid: "59b3f406e90e41da95eeea9034e35f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be25b53ecd6462ba0421e66bb203ca3"
I20260928 17:49:08.718338 18491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7e327fb27942548b10cb0b5f02fa5f" candidate_uuid: "59b3f406e90e41da95eeea9034e35f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316afb7358804495bcc19e323a81c98f"
I20260928 17:49:08.718386 18623 raft_consensus.cc:3037] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:08.718400 18491 raft_consensus.cc:3037] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:08.718910 18623 raft_consensus.cc:2454] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b3f406e90e41da95eeea9034e35f2e in term 1.
I20260928 17:49:08.718954 18491 raft_consensus.cc:2454] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b3f406e90e41da95eeea9034e35f2e in term 1.
I20260928 17:49:08.719100 18285 leader_election.cc:302] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316afb7358804495bcc19e323a81c98f, 59b3f406e90e41da95eeea9034e35f2e; no voters:
I20260928 17:49:08.719192 18404 raft_consensus.cc:2781] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:49:08.719339 18404 raft_consensus.cc:686] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 LEADER]: Becoming Leader. State: Replica: 59b3f406e90e41da95eeea9034e35f2e, State: Running, Role: LEADER
I20260928 17:49:08.719395 18404 consensus_queue.cc:237] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:08.719983 18210 catalog_manager.cc:5495] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59b3f406e90e41da95eeea9034e35f2e (127.3.222.1). New cstate: current_term: 1 leader_uuid: "59b3f406e90e41da95eeea9034e35f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } health_report { overall_health: HEALTHY } } }
I20260928 17:49:08.730216 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.728120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.3.222.1:45207)
0928 17:49:08.728168 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:08.728170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:08.728176 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:49:08.728201 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:08.728202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:08.728215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:08.728270 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:49:08.728364 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.728813 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.729505 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:08.729702 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:08.729754 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:08.729918 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:49:08.730009 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:49:08.730068 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:08.730138 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 17:49:08.730159 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 17:49:08.731552 18623 raft_consensus.cc:1261] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 1 FOLLOWER]: Refusing update from remote peer 59b3f406e90e41da95eeea9034e35f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:08.731568 18491 raft_consensus.cc:1261] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 1 FOLLOWER]: Refusing update from remote peer 59b3f406e90e41da95eeea9034e35f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:08.731864 18685 consensus_queue.cc:1035] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:08.732002 18404 consensus_queue.cc:1035] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:08.734411 18690 mvcc.cc:205] Tried to move back new op lower bound from 7334370298802073600 to 7334370298755096576. Current Snapshot: MvccSnapshot[applied={T|T < 7334370298802073600}]
I20260928 17:49:08.735844 18209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:08.735921 18209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:08.736699 18209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Sending DeleteTablet for 3 replicas of tablet cc7e327fb27942548b10cb0b5f02fa5f
I20260928 17:49:08.736938 18332 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7e327fb27942548b10cb0b5f02fa5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:08 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:33608
I20260928 17:49:08.737010 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 18248
I20260928 17:49:08.737038 18471 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7e327fb27942548b10cb0b5f02fa5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:08 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:55074
I20260928 17:49:08.737497 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7e327fb27942548b10cb0b5f02fa5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:08 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:55736
I20260928 17:49:08.737685 18698 tablet_replica.cc:330] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: stopping tablet replica
I20260928 17:49:08.737778 18698 raft_consensus.cc:2229] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:49:08.737870 18698 raft_consensus.cc:2258] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:08.738211 18698 ts_tablet_manager.cc:1905] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:08.739745 18698 ts_tablet_manager.cc:1918] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:49:08.739820 18698 log.cc:1196] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/cc7e327fb27942548b10cb0b5f02fa5f
I20260928 17:49:08.740024 18698 ts_tablet_manager.cc:1939] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Deleting consensus metadata
I20260928 17:49:08.740357 18195 catalog_manager.cc:4841] TS 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243): tablet cc7e327fb27942548b10cb0b5f02fa5f (table test-table [id=00836448daed40a0b6e95cd1841435b6]) successfully deleted
I20260928 17:49:08.742724 18697 tablet_replica.cc:330] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: stopping tablet replica
I20260928 17:49:08.742749 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 18403
W20260928 17:49:08.743564 18194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:45207 (error 108)
I20260928 17:49:08.743965 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.743854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:45207 (local address 127.0.0.1:33634)
0928 17:49:08.743899 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.743935 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:45207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 17:49:08.744601 18194 catalog_manager.cc:4584] TS 59b3f406e90e41da95eeea9034e35f2e (127.3.222.1:45207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc7e327fb27942548b10cb0b5f02fa5f: Network error: Client connection negotiation failed: client connection to 127.3.222.1:45207: connect: Connection refused (error 111)
I20260928 17:49:08.747483 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 18540
I20260928 17:49:08.747562 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:08.747459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40479 (local address 127.0.0.1:55088)
0928 17:49:08.747509 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:08.747534 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.2:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:49:08.751529 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 18177
I20260928 17:49:08.755612 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38581
--webserver_interface=127.3.222.62
--webserver_port=34285
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:38581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:08.831651 18700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:08.831835 18700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:08.831854 18700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:08.832955 18700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:49:08.832991 18700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:08.833004 18700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:08.833015 18700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:49:08.833038 18700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:49:08.834420 18700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:38581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=34285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:08.834630 18700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:08.834822 18700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:08.839640 18705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:08.839670 18708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:08.839720 18706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.839726 18700 server_base.cc:1034] running on GCE node
I20260928 17:49:08.840023 18700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:08.840279 18700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:08.841420 18700 hybrid_clock.cc:648] HybridClock initialized: now 1790617748841407 us; error 38 us; skew 500 ppm
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:08.843704 18700 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:49:08.844120 18700 webserver.cc:460] Webserver started at http://127.3.222.62:34285/ using document root <none> and password file <none>
I20260928 17:49:08.844305 18700 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:08.844350 18700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:08.845559 18700 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:49:08.846163 18715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.846336 18700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.846413 18700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "c900a3e55b2447db96c4721ca6f6d142"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "9f4bdfb6a37cc2613a92f1458f9ee9f3"
server_key_iv: "d82e5ef7bb2d2e70ae432950f0b9a7a2"
server_key_version: "encryptionkey@0"
I20260928 17:49:08.846712 18700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:08.881476 18700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:08.883234 18700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:08.883371 18700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:08.886902 18700 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:38581
I20260928 17:49:08.886940 18767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:38581 every 8 connection(s)
I20260928 17:49:08.887283 18700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:49:08.888720 18768 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:49:08.889315 18768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Bootstrap starting.
I20260928 17:49:08.889801 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 18700
I20260928 17:49:08.890051 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:45207
--local_ip_for_outbound_sockets=127.3.222.1
--tserver_master_addrs=127.3.222.62:38581
--webserver_port=36859
--webserver_interface=127.3.222.1
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:08.896972 18768 log.cc:824] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:08.898926 18768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:49:08.899160 18768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142: Bootstrap complete.
I20260928 17:49:08.901165 18768 raft_consensus.cc:348] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:08.901371 18768 raft_consensus.cc:729] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c900a3e55b2447db96c4721ca6f6d142, State: Initialized, Role: FOLLOWER
I20260928 17:49:08.901511 18768 consensus_queue.cc:260] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:08.901588 18768 raft_consensus.cc:388] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:49:08.901636 18768 raft_consensus.cc:482] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:49:08.901681 18768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 2 FOLLOWER]: Advancing to term 3
I20260928 17:49:08.902592 18768 raft_consensus.cc:504] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:08.902681 18768 leader_election.cc:302] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c900a3e55b2447db96c4721ca6f6d142; no voters:
I20260928 17:49:08.902822 18768 leader_election.cc:288] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 17:49:08.902882 18772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 17:49:08.903053 18768 sys_catalog.cc:564] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:49:08.903086 18772 raft_consensus.cc:686] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [term 3 LEADER]: Becoming Leader. State: Replica: c900a3e55b2447db96c4721ca6f6d142, State: Running, Role: LEADER
I20260928 17:49:08.903182 18772 consensus_queue.cc:237] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } }
I20260928 17:49:08.903477 18773 sys_catalog.cc:455] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c900a3e55b2447db96c4721ca6f6d142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } } }
I20260928 17:49:08.903548 18773 sys_catalog.cc:458] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:08.903680 18774 sys_catalog.cc:455] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c900a3e55b2447db96c4721ca6f6d142. Latest consensus state: current_term: 3 leader_uuid: "c900a3e55b2447db96c4721ca6f6d142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c900a3e55b2447db96c4721ca6f6d142" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38581 } } }
I20260928 17:49:08.903728 18774 sys_catalog.cc:458] T 00000000000000000000000000000000 P c900a3e55b2447db96c4721ca6f6d142 [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:08.903987 18779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:49:08.904755 18779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:49:08.904882 18779 catalog_manager.cc:1259] Loaded cluster ID: 50eaabda8e124023b332028b8a5fa835
I20260928 17:49:08.904918 18779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 17:49:08.905259 18779 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f903851a9ad google::LogMessage::Fail() at ??:0
@ 0x7f903851d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f903851a4de google::LogMessage::Flush() at ??:0
@ 0x7f903851ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f904043f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f90404488b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f903a3e80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f903a3dd954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f903b2416db start_thread at ??:0
@ 0x7f903701861f clone at ??:0
W20260928 17:49:08.978520 18770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:08.978700 18770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:08.978722 18770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:08.979878 18770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:08.979920 18770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:08.979955 18770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:49:08.981441 18770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:45207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=36859
--tserver_master_addrs=127.3.222.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:08.981659 18770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:08.981848 18770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:08.986450 18805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.986627 18770 server_base.cc:1034] running on GCE node
W20260928 17:49:08.986474 18807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:08.986446 18804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:08.986914 18770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:08.987125 18770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:08.988260 18770 hybrid_clock.cc:648] HybridClock initialized: now 1790617748988255 us; error 39 us; skew 500 ppm
Sep 28 17:49:08 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:08.990477 18770 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:49:08.990866 18770 webserver.cc:460] Webserver started at http://127.3.222.1:36859/ using document root <none> and password file <none>
I20260928 17:49:08.991062 18770 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:08.991109 18770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:08.992377 18770 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:49:08.993013 18814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:08.993162 18770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:08.993238 18770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "59b3f406e90e41da95eeea9034e35f2e"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "e1eb72288d0e1a45b012a0718d774d7b"
server_key_iv: "beb253e73d407a121291e39cb67d29bf"
server_key_version: "encryptionkey@0"
I20260928 17:49:08.993535 18770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:09.005613 18770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:09.007725 18770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:09.007859 18770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:09.008054 18770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:09.008441 18821 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:49:09.010285 18770 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:49:09.010330 18770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:49:09.010363 18770 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:49:09.010953 18770 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:49:09.010985 18770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 17:49:09.011072 18821 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Bootstrap starting.
I20260928 17:49:09.017472 18770 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:45207
I20260928 17:49:09.017612 18928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:45207 every 8 connection(s)
I20260928 17:49:09.017860 18770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:49:09.021868 18821 log.cc:824] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:09.024442 18821 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:49:09.024549 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 18770
I20260928 17:49:09.024732 18821 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Bootstrap complete.
I20260928 17:49:09.024806 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:40479
--local_ip_for_outbound_sockets=127.3.222.2
--tserver_master_addrs=127.3.222.62:38581
--webserver_port=41347
--webserver_interface=127.3.222.2
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:09.025786 18821 ts_tablet_manager.cc:1397] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260928 17:49:09.027341 18821 raft_consensus.cc:348] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:09.027688 18821 raft_consensus.cc:729] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3f406e90e41da95eeea9034e35f2e, State: Initialized, Role: FOLLOWER
I20260928 17:49:09.027843 18821 consensus_queue.cc:260] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:09.028090 18821 ts_tablet_manager.cc:1428] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260928 17:49:09.079182 18931 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:49:09.018464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:38613)
0928 17:49:09.018618 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.018658 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:09.019096 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:09.079004 (+ 59908us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
W20260928 17:49:09.079399 18929 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:38581 (0 consecutive failures): Network error: Failed to ping master at 127.3.222.62:38581: Client connection negotiation failed: client connection to 127.3.222.62:38581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:49:09.079735 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.079629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:49225)
0928 17:49:09.079675 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.079715 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 17:49:09.080051 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.079969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:43841)
0928 17:49:09.080008 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.080037 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 17:49:09.080149 18929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:49:09.103704 18933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:09.103888 18933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:09.103909 18933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:09.104980 18933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:09.105014 18933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:09.105038 18933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:49:09.106384 18933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:40479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=41347
--tserver_master_addrs=127.3.222.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:09.106555 18933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:09.106720 18933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:09.111268 18939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:09.111287 18940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:09.111268 18942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:09.111521 18933 server_base.cc:1034] running on GCE node
I20260928 17:49:09.111717 18933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:09.111927 18933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:09.113093 18933 hybrid_clock.cc:648] HybridClock initialized: now 1790617749113074 us; error 31 us; skew 500 ppm
Sep 28 17:49:09 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:09.115355 18933 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:49:09.115728 18933 webserver.cc:460] Webserver started at http://127.3.222.2:41347/ using document root <none> and password file <none>
I20260928 17:49:09.115917 18933 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:09.115962 18933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:09.117223 18933 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:49:09.117834 18949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:09.118000 18933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:09.118090 18933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "316afb7358804495bcc19e323a81c98f"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "799b9b662c1bb2884210a68506416635"
server_key_iv: "5f211170f782ae5821809d8bc7318e1b"
server_key_version: "encryptionkey@0"
I20260928 17:49:09.118408 18933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:09.129158 18933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:09.131033 18933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:09.131153 18933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:09.131362 18933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:09.131752 18956 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:49:09.133096 18933 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:49:09.133136 18933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:49:09.133167 18933 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:49:09.133750 18933 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:49:09.133780 18933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 17:49:09.133838 18956 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Bootstrap starting.
I20260928 17:49:09.140208 18933 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:40479
I20260928 17:49:09.140364 19063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:40479 every 8 connection(s)
I20260928 17:49:09.140589 18933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:49:09.141657 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.141186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:36071)
0928 17:49:09.141365 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.141468 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
W20260928 17:49:09.141866 19064 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:38581 (0 consecutive failures): Network error: Failed to ping master at 127.3.222.62:38581: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
I20260928 17:49:09.142153 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.142046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:56181)
0928 17:49:09.142093 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.142127 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:49:09.142421 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.142336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:46533)
0928 17:49:09.142374 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.142400 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 17:49:09.142516 19064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 17:49:09.142781 18956 log.cc:824] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:09.144852 18956 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:49:09.145067 18956 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Bootstrap complete.
I20260928 17:49:09.145763 18956 ts_tablet_manager.cc:1397] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260928 17:49:09.146869 18956 raft_consensus.cc:348] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:09.147110 18956 raft_consensus.cc:729] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316afb7358804495bcc19e323a81c98f, State: Initialized, Role: FOLLOWER
I20260928 17:49:09.147228 18956 consensus_queue.cc:260] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:09.147431 18956 ts_tablet_manager.cc:1428] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260928 17:49:09.147722 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.147613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:38733)
0928 17:49:09.147671 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.147701 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 17:49:09.147948 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.147885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:36453)
0928 17:49:09.147917 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.147935 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 17:49:09.149336 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 18933
I20260928 17:49:09.149573 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:40243
--local_ip_for_outbound_sockets=127.3.222.3
--tserver_master_addrs=127.3.222.62:38581
--webserver_port=38643
--webserver_interface=127.3.222.3
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:09.227113 19069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:09.227310 19069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:09.227331 19069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:09.228407 19069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:09.228446 19069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:09.228470 19069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:49:09.229856 19069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:40243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=38643
--tserver_master_addrs=127.3.222.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:09.230057 19069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:09.230255 19069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:09.234767 19077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:09.234804 19075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:09.234845 19074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:09.234889 19069 server_base.cc:1034] running on GCE node
I20260928 17:49:09.235069 19069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:09.235319 19069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:09.236459 19069 hybrid_clock.cc:648] HybridClock initialized: now 1790617749236423 us; error 50 us; skew 500 ppm
Sep 28 17:49:09 dist-test-slave-w3bm krb5kdc[17633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:09.238772 19069 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:49:09.239140 19069 webserver.cc:460] Webserver started at http://127.3.222.3:38643/ using document root <none> and password file <none>
I20260928 17:49:09.239351 19069 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:09.239395 19069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:09.240617 19069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:49:09.241156 19084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:09.241317 19069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:09.241385 19069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "9be25b53ecd6462ba0421e66bb203ca3"
format_stamp: "Formatted at 2026-09-28 17:49:07 on dist-test-slave-w3bm"
server_key: "c7fe643248ccb5540da5e6fecfceebd3"
server_key_iv: "2f95d15ad072f1f532d1c6752fbd4f4f"
server_key_version: "encryptionkey@0"
I20260928 17:49:09.241706 19069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:09.266010 19069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:09.268100 19069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:09.268221 19069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:09.268488 19069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:09.268972 19069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:09.269012 19069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:09.269032 19069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:09.269047 19069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:09.274374 19069 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:40243
I20260928 17:49:09.274418 19197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:40243 every 8 connection(s)
I20260928 17:49:09.274730 19069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:49:09.275707 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.275339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:33247)
0928 17:49:09.275505 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.275573 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
W20260928 17:49:09.276033 19198 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:38581 (0 consecutive failures): Network error: Failed to ping master at 127.3.222.62:38581: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
I20260928 17:49:09.276322 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.276223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:40251)
0928 17:49:09.276276 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.276303 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 17:49:09.276553 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:09.276486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:55297)
0928 17:49:09.276519 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:09.276539 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 17:49:09.276628 19198 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 17:49:09.284072 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 19069
I20260928 17:49:10.080756 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.080514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:49067)
0928 17:49:10.080654 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:10.080695 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260928 17:49:10.148438 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.148252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:48459)
0928 17:49:10.148353 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:10.148386 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260928 17:49:10.211580 19204 raft_consensus.cc:482] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:49:10.211707 19204 raft_consensus.cc:504] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:10.212069 19204 leader_election.cc:288] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316afb7358804495bcc19e323a81c98f (127.3.222.2:40479), 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243)
Sep 28 17:49:10 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.3@KRBTEST.COM
Sep 28 17:49:10 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.2@KRBTEST.COM
I20260928 17:49:10.223321 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.212361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:41489 (local address 127.3.222.2:40479)
0928 17:49:10.212507 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:10.212519 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:10.212542 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:49:10.212671 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:10.212674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:10.212723 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:10.212851 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:10.213635 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:10.215033 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:10.216533 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:10.217172 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:10.217280 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:10.220304 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:10.220308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:10.220309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:10.220321 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:10.221963 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:10.222311 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:10.222314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:10.222315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:10.222405 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:10.222576 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:10.222576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:10.222577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:10.222795 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:10.222869 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:10.223060 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 17:49:10.223185 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":120,"threads_started":1}
I20260928 17:49:10.223321 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.212435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:40865 (local address 127.3.222.3:40243)
0928 17:49:10.212594 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:10.212601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:10.212619 (+ 18us) server_negotiation.cc:407] Connection header received
0928 17:49:10.213111 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:10.213114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:10.213165 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:10.213263 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:10.213629 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:10.215030 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:10.216556 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:10.217170 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:10.217280 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:10.219862 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:10.219866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:10.219867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:10.219880 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:10.221609 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:10.222075 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:10.222078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:10.222079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:10.222179 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:10.222427 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:10.222427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:10.222428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:10.222718 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:10.222774 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:10.223068 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 17:49:10.223190 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":51,"threads_started":1}
I20260928 17:49:10.223340 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.212218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:40479 (local address 127.3.222.1:41489)
0928 17:49:10.212296 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:10.212309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:10.212378 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:10.212914 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:10.212917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:10.213217 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:10.213477 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:10.213484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:10.215139 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:10.215140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:10.216430 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:10.216433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:10.217266 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:10.217267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:10.217409 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:10.218062 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:10.218067 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:10.220191 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:10.222047 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:10.222048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:10.222049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:10.222246 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:10.222467 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:10.222467 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:10.222468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:10.222523 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:10.222859 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:10.222860 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:10.222919 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 17:49:10.223002 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 17:49:10.223200 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":61}
I20260928 17:49:10.223855 19018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7e327fb27942548b10cb0b5f02fa5f" candidate_uuid: "59b3f406e90e41da95eeea9034e35f2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "316afb7358804495bcc19e323a81c98f" is_pre_election: true
I20260928 17:49:10.223991 19018 raft_consensus.cc:2454] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b3f406e90e41da95eeea9034e35f2e in term 1.
I20260928 17:49:10.224038 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.212329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:40243 (local address 127.3.222.1:40865)
0928 17:49:10.212460 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:10.212470 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:10.212529 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:10.213290 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:10.213291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:10.213429 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:10.213477 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:10.213484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:10.215142 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:10.215143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:10.216430 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:10.216433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:10.217265 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:10.217266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:10.217409 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:10.218053 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:10.218063 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:10.219757 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:10.221708 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:10.221715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:10.221717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:10.221986 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:10.222246 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:10.222246 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:10.222247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:10.222369 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:10.222779 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:10.222785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:10.222919 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 17:49:10.223002 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 17:49:10.223288 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":183,"thread_start_us":60,"threads_started":1}
I20260928 17:49:10.224299 18816 leader_election.cc:302] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316afb7358804495bcc19e323a81c98f, 59b3f406e90e41da95eeea9034e35f2e; no voters:
I20260928 17:49:10.224403 19204 raft_consensus.cc:2781] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 17:49:10.224452 19204 raft_consensus.cc:482] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:49:10.224473 19204 raft_consensus.cc:3037] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 1 FOLLOWER]: Advancing to term 2
W20260928 17:49:10.224660 18816 leader_election.cc:341] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243): Not found: Tablet not found: cc7e327fb27942548b10cb0b5f02fa5f
I20260928 17:49:10.225589 19204 raft_consensus.cc:504] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:10.225703 19204 leader_election.cc:288] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 2 election: Requested vote from peers 316afb7358804495bcc19e323a81c98f (127.3.222.2:40479), 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243)
I20260928 17:49:10.225860 19018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7e327fb27942548b10cb0b5f02fa5f" candidate_uuid: "59b3f406e90e41da95eeea9034e35f2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "316afb7358804495bcc19e323a81c98f"
I20260928 17:49:10.225925 19018 raft_consensus.cc:3037] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 1 FOLLOWER]: Advancing to term 2
W20260928 17:49:10.225939 18816 leader_election.cc:341] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243): Not found: Tablet not found: cc7e327fb27942548b10cb0b5f02fa5f
I20260928 17:49:10.226892 19018 raft_consensus.cc:2454] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b3f406e90e41da95eeea9034e35f2e in term 2.
I20260928 17:49:10.227070 18816 leader_election.cc:302] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316afb7358804495bcc19e323a81c98f, 59b3f406e90e41da95eeea9034e35f2e; no voters: 9be25b53ecd6462ba0421e66bb203ca3
I20260928 17:49:10.227154 19204 raft_consensus.cc:2781] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:49:10.227298 19204 raft_consensus.cc:686] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [term 2 LEADER]: Becoming Leader. State: Replica: 59b3f406e90e41da95eeea9034e35f2e, State: Running, Role: LEADER
I20260928 17:49:10.227380 19204 consensus_queue.cc:237] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:10.227573 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.227480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:44375)
0928 17:49:10.227523 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:10.227548 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:49:10.277081 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.276886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:35821)
0928 17:49:10.277014 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:10.277049 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260928 17:49:10.754796 19018 raft_consensus.cc:1261] T cc7e327fb27942548b10cb0b5f02fa5f P 316afb7358804495bcc19e323a81c98f [term 2 FOLLOWER]: Refusing update from remote peer 59b3f406e90e41da95eeea9034e35f2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 17:49:10.755132 19211 consensus_queue.cc:1035] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 17:49:10.756038 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.755888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:47581)
0928 17:49:10.755966 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:10.756002 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":38,"threads_started":1}
I20260928 17:49:10.756855 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.756708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:43113)
0928 17:49:10.756790 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:10.756817 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":53,"threads_started":1}
W20260928 17:49:10.756901 18816 consensus_peers.cc:487] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e -> Peer 9be25b53ecd6462ba0421e66bb203ca3 (127.3.222.3:40243): Couldn't send request to peer 9be25b53ecd6462ba0421e66bb203ca3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc7e327fb27942548b10cb0b5f02fa5f. This is attempt 1: this message will repeat every 5th retry.
I20260928 17:49:10.756992 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:10.756912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:59353)
0928 17:49:10.756944 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:10.756968 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 17:49:11.277645 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:11.277467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:51357)
0928 17:49:11.277575 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:11.277609 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260928 17:49:11.378835 19221 ts_tablet_manager.cc:927] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Initiating tablet copy from peer 59b3f406e90e41da95eeea9034e35f2e (127.3.222.1:45207)
I20260928 17:49:11.379091 19221 tablet_copy_client.cc:316] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.222.1:45207
Sep 28 17:49:11 dist-test-slave-w3bm krb5kdc[17633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:49:11.386513 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:11.379398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:45207 (local address 127.3.222.3:46139)
0928 17:49:11.379472 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:11.379495 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:11.379574 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:11.379871 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:11.379877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:11.380179 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:11.380238 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:11.380240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:11.381157 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:11.381158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:11.381775 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:11.381777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:11.382300 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:11.382301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:11.382363 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:11.382785 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:11.382789 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:11.384055 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:11.385563 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:11.385569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:11.385571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:11.385748 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:11.385950 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:11.385951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:11.385951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:11.386015 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:11.386243 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:11.386249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:11.386292 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 17:49:11.386353 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 17:49:11.386444 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 17:49:11.386521 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:11.379525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:46139 (local address 127.3.222.1:45207)
0928 17:49:11.379703 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:11.379708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:11.379734 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:49:11.379758 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:11.379760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:11.379773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:11.379837 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:11.380302 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:11.381093 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:11.381853 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:11.382241 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:11.382296 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:11.384126 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:11.384133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:11.384134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:11.384140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:11.385506 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:11.385815 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:11.385817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:11.385818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:11.385869 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:11.386072 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:11.386072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:11.386073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:11.386199 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:11.386255 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:11.386382 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 17:49:11.386441 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260928 17:49:11.386971 18903 tablet_copy_service.cc:140] P 59b3f406e90e41da95eeea9034e35f2e: Received BeginTabletCopySession request for tablet cc7e327fb27942548b10cb0b5f02fa5f from peer 9be25b53ecd6462ba0421e66bb203ca3 ({username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:46139)
I20260928 17:49:11.387040 18903 tablet_copy_service.cc:161] P 59b3f406e90e41da95eeea9034e35f2e: Beginning new tablet copy session on tablet cc7e327fb27942548b10cb0b5f02fa5f from peer 9be25b53ecd6462ba0421e66bb203ca3 at {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:46139: session id = 9be25b53ecd6462ba0421e66bb203ca3-cc7e327fb27942548b10cb0b5f02fa5f
I20260928 17:49:11.387630 18903 tablet_copy_source_session.cc:205] T cc7e327fb27942548b10cb0b5f02fa5f P 59b3f406e90e41da95eeea9034e35f2e: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 17:49:11.388567 19221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7e327fb27942548b10cb0b5f02fa5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:11.391027 19221 tablet_copy_client.cc:791] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: tablet copy: Starting download of 0 data blocks...
I20260928 17:49:11.391149 19221 tablet_copy_client.cc:655] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: tablet copy: Starting download of 1 WAL segments...
I20260928 17:49:11.391975 19221 tablet_copy_client.cc:531] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 17:49:11.392757 19221 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Bootstrap starting.
I20260928 17:49:11.394411 19221 log.cc:824] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:11.396291 19221 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:49:11.396476 19221 tablet_bootstrap.cc:492] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Bootstrap complete.
I20260928 17:49:11.396589 19221 ts_tablet_manager.cc:1397] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 17:49:11.397495 19221 raft_consensus.cc:348] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:11.397634 19221 raft_consensus.cc:729] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be25b53ecd6462ba0421e66bb203ca3, State: Initialized, Role: FOLLOWER
I20260928 17:49:11.397744 19221 consensus_queue.cc:260] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316afb7358804495bcc19e323a81c98f" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 40479 } } peers { permanent_uuid: "9be25b53ecd6462ba0421e66bb203ca3" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 40243 } } peers { permanent_uuid: "59b3f406e90e41da95eeea9034e35f2e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 45207 } }
I20260928 17:49:11.397998 19221 ts_tablet_manager.cc:1428] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260928 17:49:11.398240 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:11.398140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:36821)
0928 17:49:11.398190 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:11.398219 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 17:49:11.398250 18903 tablet_copy_service.cc:342] P 59b3f406e90e41da95eeea9034e35f2e: Request end of tablet copy session 9be25b53ecd6462ba0421e66bb203ca3-cc7e327fb27942548b10cb0b5f02fa5f received from {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:46139
I20260928 17:49:11.398303 18903 tablet_copy_service.cc:433] P 59b3f406e90e41da95eeea9034e35f2e: ending tablet copy session 9be25b53ecd6462ba0421e66bb203ca3-cc7e327fb27942548b10cb0b5f02fa5f on tablet cc7e327fb27942548b10cb0b5f02fa5f with peer 9be25b53ecd6462ba0421e66bb203ca3
I20260928 17:49:11.398523 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:11.398440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:47877)
0928 17:49:11.398471 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:11.398496 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 17:49:11.757462 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:11.757297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:44787)
0928 17:49:11.757389 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:11.757417 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":44,"threads_started":1}
I20260928 17:49:11.757484 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:11.757295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:51331)
0928 17:49:11.757410 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:11.757444 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260928 17:49:11.996526 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:11.996390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:43283)
0928 17:49:11.996441 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:11.996477 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:49:11.996555 19152 raft_consensus.cc:1203] T cc7e327fb27942548b10cb0b5f02fa5f P 9be25b53ecd6462ba0421e66bb203ca3 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 17:49:11.997097 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:11.996894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:48091)
0928 17:49:11.997021 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:11.997045 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":47,"threads_started":1}
I20260928 17:49:12.758060 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:12.757836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:54651)
0928 17:49:12.757969 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:12.758005 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260928 17:49:12.997120 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:12.996891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:56769)
0928 17:49:12.997007 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:12.997040 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260928 17:49:12.997598 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:12.997419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:41725)
0928 17:49:12.997530 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:12.997562 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260928 17:49:13.758674 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:13.758461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:56215)
0928 17:49:13.758578 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:13.758616 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260928 17:49:13.997702 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:13.997486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:47723)
0928 17:49:13.997618 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:13.997654 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260928 17:49:13.998150 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:13.997991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:57201)
0928 17:49:13.998084 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:13.998112 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":44,"threads_started":1}
I20260928 17:49:14.759351 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:14.759127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:57345)
0928 17:49:14.759273 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:14.759306 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260928 17:49:14.998327 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:14.998127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:49981)
0928 17:49:14.998247 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:14.998279 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260928 17:49:14.998710 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:14.998607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:38139)
0928 17:49:14.998655 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:14.998682 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 17:49:14.998824 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:14.998644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:58921)
0928 17:49:14.998750 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:14.998784 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
I20260928 17:49:15.759995 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:15.759776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:39393)
0928 17:49:15.759888 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:15.759923 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260928 17:49:15.999320 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:15.999089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:44321)
0928 17:49:15.999236 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:15.999273 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260928 17:49:15.999382 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:15.999195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:40481)
0928 17:49:15.999311 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:15.999342 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260928 17:49:16.760601 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:16.760407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:47257)
0928 17:49:16.760520 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:16.760555 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260928 17:49:16.999929 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:16.999725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:47313)
0928 17:49:16.999849 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:16.999881 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 17:49:16.999938 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:16.999726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:54963)
0928 17:49:16.999857 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:16.999892 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260928 17:49:17.761197 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:17.760996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:44925)
0928 17:49:17.761124 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:17.761152 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260928 17:49:18.000506 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:18.000320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:56709)
0928 17:49:18.000431 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:18.000461 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260928 17:49:18.000519 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:18.000329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:49041)
0928 17:49:18.000450 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:18.000477 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260928 17:49:18.761806 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:18.761590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:49847)
0928 17:49:18.761708 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:18.761750 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260928 17:49:19.001119 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:19.000915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:39335)
0928 17:49:19.001036 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:19.001072 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260928 17:49:19.001138 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:19.000915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:58845)
0928 17:49:19.001067 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:19.001096 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260928 17:49:19.762420 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:19.762227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:44753)
0928 17:49:19.762344 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:19.762377 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260928 17:49:20.001760 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:20.001567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:59003)
0928 17:49:20.001679 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:20.001712 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 17:49:20.001760 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:20.001534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:52391)
0928 17:49:20.001671 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:20.001714 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260928 17:49:20.763041 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:20.762808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:44323)
0928 17:49:20.762962 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:20.762994 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260928 17:49:21.002377 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:21.002142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:42467)
0928 17:49:21.002294 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:21.002330 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260928 17:49:21.002381 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:21.002200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:44525)
0928 17:49:21.002308 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:21.002339 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260928 17:49:21.763751 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:21.763509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:54079)
0928 17:49:21.763651 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:21.763696 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260928 17:49:22.002995 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:22.002796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:51057)
0928 17:49:22.002915 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:22.002947 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260928 17:49:22.003068 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:22.002870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:59747)
0928 17:49:22.002987 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:22.003020 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260928 17:49:22.764420 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:22.764214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:58965)
0928 17:49:22.764336 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:22.764371 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260928 17:49:23.003647 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:23.003429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:40741)
0928 17:49:23.003545 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:23.003597 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260928 17:49:23.003645 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:23.003452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:49129)
0928 17:49:23.003570 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:23.003602 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":52,"threads_started":1}
I20260928 17:49:23.765021 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:23.764809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.2:54701)
0928 17:49:23.764939 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:23.764976 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260928 17:49:24.004338 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:24.004120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.3:33331)
0928 17:49:24.004254 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:24.004291 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260928 17:49:24.004338 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:24.004120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38581 (local address 127.3.222.1:55639)
0928 17:49:24.004267 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:24.004294 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260928 17:49:24.289786 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 18770
I20260928 17:49:24.294106 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 18933
I20260928 17:49:24.298005 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 19069
2026-09-28T17:49:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17935 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:49:24 dist-test-slave-w3bm krb5kdc[19270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:49:24 dist-test-slave-w3bm krb5kdc[19270](info): set up 2 sockets
Sep 28 17:49:24 dist-test-slave-w3bm krb5kdc[19270](info): commencing operation
krb5kdc: starting...
W20260928 17:49:25.262946 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:49:25 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:49:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:49:25.378829 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36989
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:36989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:25.456501 19283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:25.456694 19283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:25.456723 19283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:25.457814 19283 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 17:49:25.457851 19283 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 17:49:25.457867 19283 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 17:49:25.457881 19283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:49:25.457907 19283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:25.457933 19283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:25.457957 19283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:49:25.457979 19283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:49:25.459443 19283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:36989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:25.459663 19283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:25.459851 19283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:49:25.464421 19283 server_base.cc:1034] running on GCE node
W20260928 17:49:25.464344 19288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:25.464361 19289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:25.464356 19291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:25.464859 19283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:25.465147 19283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:25.466298 19283 hybrid_clock.cc:648] HybridClock initialized: now 1790617765466279 us; error 33 us; skew 500 ppm
Sep 28 17:49:25 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:25.468650 19283 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:49:25.469074 19283 webserver.cc:460] Webserver started at http://127.3.222.62:36263/ using document root <none> and password file <none>
I20260928 17:49:25.469249 19283 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:25.469285 19283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:25.469374 19283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:25.470364 19283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "e90fc26731db4c4bb93d09cd61f522ca"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "efc4c0c80149a11af7d1526b771886a9"
server_key_iv: "6b61babd9e4ffa7cd0576d6e48a1f90f"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.470655 19283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "e90fc26731db4c4bb93d09cd61f522ca"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "efc4c0c80149a11af7d1526b771886a9"
server_key_iv: "6b61babd9e4ffa7cd0576d6e48a1f90f"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.471855 19283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:49:25.472595 19298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.472760 19283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:49:25.472852 19283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "e90fc26731db4c4bb93d09cd61f522ca"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "efc4c0c80149a11af7d1526b771886a9"
server_key_iv: "6b61babd9e4ffa7cd0576d6e48a1f90f"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.472911 19283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:25.489462 19283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:25.491107 19283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:25.491279 19283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:25.495013 19283 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:36989
I20260928 17:49:25.495043 19350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:36989 every 8 connection(s)
I20260928 17:49:25.495369 19283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:49:25.496217 19351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:25.498401 19351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Bootstrap starting.
I20260928 17:49:25.498943 19351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:25.499228 19351 log.cc:824] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:25.499831 19351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: No bootstrap required, opened a new log
I20260928 17:49:25.501353 19351 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:49:25.501487 19351 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:25.501508 19351 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90fc26731db4c4bb93d09cd61f522ca, State: Initialized, Role: FOLLOWER
I20260928 17:49:25.501646 19351 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:49:25.501713 19351 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:49:25.501734 19351 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:49:25.501778 19351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:25.502270 19351 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:49:25.502357 19351 leader_election.cc:302] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90fc26731db4c4bb93d09cd61f522ca; no voters:
I20260928 17:49:25.502511 19351 leader_election.cc:288] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:49:25.502573 19354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:49:25.502731 19354 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 1 LEADER]: Becoming Leader. State: Replica: e90fc26731db4c4bb93d09cd61f522ca, State: Running, Role: LEADER
I20260928 17:49:25.502761 19351 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:49:25.502820 19354 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:49:25.503302 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 19283
I20260928 17:49:25.503417 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:49:25.503455 19355 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90fc26731db4c4bb93d09cd61f522ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } } }
I20260928 17:49:25.503535 19355 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:25.503612 3960 external_mini_cluster.cc:1428] Setting key c5eeeae22b638b30ddfb78415d32ac83
I20260928 17:49:25.503940 19363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:49:25.504496 19356 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90fc26731db4c4bb93d09cd61f522ca. Latest consensus state: current_term: 1 leader_uuid: "e90fc26731db4c4bb93d09cd61f522ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } } }
I20260928 17:49:25.504535 19363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:49:25.504609 19356 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:25.506376 19363 catalog_manager.cc:1347] Generated new cluster ID: cfc818a997094d9d9cd7297d97192294
I20260928 17:49:25.506444 19363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:49:25 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:25.514951 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:25.504181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.3.222.62:36989)
0928 17:49:25.504909 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:25.504918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:25.504944 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:49:25.505034 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:25.505039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:25.505080 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:25.505201 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:25.505922 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.507160 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.508255 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.508826 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.508917 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:25.511506 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:25.511510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:25.511510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:25.511522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:25.513369 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:25.513715 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:25.513718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:25.513718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:25.513819 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:25.513985 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:25.513985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:25.513986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:25.514289 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:25.514423 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:25.514493 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 17:49:25.514725 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":77,"threads_started":1}
I20260928 17:49:25.519531 19363 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:49:25.520192 19363 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:49:25.525949 19363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Generated new TSK 0
I20260928 17:49:25.526101 19363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:49:25.559032 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36989
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:25.661319 19377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:25.661505 19377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:25.661528 19377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:25.662648 19377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:25.662689 19377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:25.662714 19377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:49:25.664182 19377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:25.664386 19377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:25.664587 19377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:25.669480 19383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:25.669471 19382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:25.669665 19377 server_base.cc:1034] running on GCE node
W20260928 17:49:25.669813 19385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:25.669996 19377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:25.670226 19377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:25.671367 19377 hybrid_clock.cc:648] HybridClock initialized: now 1790617765671352 us; error 32 us; skew 500 ppm
Sep 28 17:49:25 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:25.673918 19377 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:49:25.674304 19377 webserver.cc:460] Webserver started at http://127.3.222.1:43367/ using document root <none> and password file <none>
I20260928 17:49:25.674481 19377 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:25.674520 19377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:25.674606 19377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:25.675638 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "ca2f00c73482424f9bd055baae51e4e7"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "1ebd54727164029558fb55bdf78e7d49"
server_key_iv: "ca8711ec4f01e757dc4ba20dfb639e72"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.675925 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "ca2f00c73482424f9bd055baae51e4e7"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "1ebd54727164029558fb55bdf78e7d49"
server_key_iv: "ca8711ec4f01e757dc4ba20dfb639e72"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.677114 19377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:49:25.677783 19392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.677922 19377 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:25.678012 19377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "ca2f00c73482424f9bd055baae51e4e7"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "1ebd54727164029558fb55bdf78e7d49"
server_key_iv: "ca8711ec4f01e757dc4ba20dfb639e72"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.678072 19377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:25.694290 19377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:25.696123 19377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:25.696249 19377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:25.696461 19377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:25.696733 19377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:25.696764 19377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.696791 19377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:25.696815 19377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.702205 19377 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:37905
I20260928 17:49:25.702255 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:37905 every 8 connection(s)
I20260928 17:49:25.702533 19377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:49:25.704063 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 19377
I20260928 17:49:25.704166 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:49:25.704371 3960 external_mini_cluster.cc:1428] Setting key 34977e585b4e28bf72d17f97dda45763
Sep 28 17:49:25 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
I20260928 17:49:25.712189 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:25.703263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:41129 (local address 127.3.222.62:36989)
0928 17:49:25.703306 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:25.703308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:25.703487 (+ 179us) server_negotiation.cc:407] Connection header received
0928 17:49:25.703998 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:25.703999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:25.704013 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:25.704059 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:25.705166 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.705699 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.706872 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.707177 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.707315 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:25.709448 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:25.709451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:25.709451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:25.709459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:25.710953 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:25.711331 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:25.711332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:25.711333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:25.711403 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:25.711610 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:25.711611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:25.711611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:25.711736 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:25.711803 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:25.712011 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 17:49:25.712099 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:49:25.712249 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:25.703224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:41129)
0928 17:49:25.703369 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:25.703409 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:25.703905 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:25.704135 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:25.704140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:25.704837 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:25.705078 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:25.705085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.705791 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.705792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:25.706787 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:25.706790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.707328 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.707329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:25.707424 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:25.707856 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:25.707864 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:25.709358 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:25.711028 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:25.711034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:25.711036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:25.711264 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:25.711459 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:25.711460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:25.711460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:25.711554 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:25.711797 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:25.711799 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:25.711902 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 17:49:25.711972 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 17:49:25.712085 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260928 17:49:25.712707 19506 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36989
I20260928 17:49:25.712788 19506 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:25.712947 19506 heartbeater.cc:511] Master 127.3.222.62:36989 requested a full tablet report, sending...
I20260928 17:49:25.713573 19315 ts_manager.cc:194] Registered new tserver with Master: ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905)
I20260928 17:49:25.714182 19315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:41129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:49:25.729274 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36989
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:25.807610 19511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:25.807797 19511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:25.807826 19511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:25.808928 19511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:25.808966 19511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:25.809000 19511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:49:25.810359 19511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:25.810580 19511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:25.810765 19511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:25.815277 19516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:25.815397 19511 server_base.cc:1034] running on GCE node
W20260928 17:49:25.815287 19517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:25.815287 19519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:25.815734 19511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:25.815953 19511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:25.817098 19511 hybrid_clock.cc:648] HybridClock initialized: now 1790617765817092 us; error 31 us; skew 500 ppm
Sep 28 17:49:25 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:25.819389 19511 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:49:25.819769 19511 webserver.cc:460] Webserver started at http://127.3.222.2:42493/ using document root <none> and password file <none>
I20260928 17:49:25.819934 19511 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:25.819971 19511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:25.820055 19511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:25.820973 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "d1f523e4cf034c049da74bc97f85cd49"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "f2531a55e2cf1648ea436f509588ccc0"
server_key_iv: "1da764b56db9ec65b8cc935c93c9767a"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.821241 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "d1f523e4cf034c049da74bc97f85cd49"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "f2531a55e2cf1648ea436f509588ccc0"
server_key_iv: "1da764b56db9ec65b8cc935c93c9767a"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.822337 19511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:49:25.822988 19526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.823117 19511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:25.823189 19511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "d1f523e4cf034c049da74bc97f85cd49"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "f2531a55e2cf1648ea436f509588ccc0"
server_key_iv: "1da764b56db9ec65b8cc935c93c9767a"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.823274 19511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:25.844925 19511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:25.846870 19511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:25.846982 19511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:25.847236 19511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:25.847568 19511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:25.847601 19511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.847625 19511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:25.847647 19511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.853127 19511 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:45401
I20260928 17:49:25.853180 19639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:45401 every 8 connection(s)
I20260928 17:49:25.853459 19511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:49:25 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:25.861811 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:25.854110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:58801 (local address 127.3.222.62:36989)
0928 17:49:25.854170 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:25.854171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:25.854347 (+ 176us) server_negotiation.cc:407] Connection header received
0928 17:49:25.854832 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:25.854833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:25.854842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:25.854898 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:25.855834 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.856351 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.857298 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.857486 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.857536 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:25.859561 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:25.859563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:25.859563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:25.859570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:25.860695 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:25.861050 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:25.861051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:25.861051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:25.861103 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:25.861309 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:25.861310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:25.861310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:25.861407 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:25.861463 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:25.861687 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 17:49:25.861749 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 17:49:25.861968 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:25.854071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:58801)
0928 17:49:25.854219 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:25.854259 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:25.854757 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:25.854951 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:25.854954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:25.855526 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:25.855745 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:25.855752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.856426 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.856427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:25.857224 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:25.857227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:25.857544 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:25.857544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:25.857653 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:25.858137 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:25.858144 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:25.859459 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:25.860778 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:25.860784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:25.860786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:25.860987 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:25.861160 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:25.861160 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:25.861161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:25.861262 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:25.861453 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:25.861456 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:25.861591 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 17:49:25.861667 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:49:25.861779 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260928 17:49:25.862406 19640 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36989
I20260928 17:49:25.862479 19640 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:25.862638 19640 heartbeater.cc:511] Master 127.3.222.62:36989 requested a full tablet report, sending...
I20260928 17:49:25.862969 19315 ts_manager.cc:194] Registered new tserver with Master: d1f523e4cf034c049da74bc97f85cd49 (127.3.222.2:45401)
I20260928 17:49:25.863337 19315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:58801
I20260928 17:49:25.863510 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 19511
I20260928 17:49:25.863595 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:49:25.863761 3960 external_mini_cluster.cc:1428] Setting key d879307fc8e53c62c069457abfa2e6ea
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:49:25.885723 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36989
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:25.961666 19645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:25.961859 19645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:25.961880 19645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:25.962994 19645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:25.963034 19645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:25.963059 19645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:49:25.964457 19645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:25.964670 19645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:25.964864 19645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:25.969493 19653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:25.969617 19651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:25.969521 19645 server_base.cc:1034] running on GCE node
W20260928 17:49:25.969494 19650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:25.969882 19645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:25.970093 19645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:25.971243 19645 hybrid_clock.cc:648] HybridClock initialized: now 1790617765971229 us; error 28 us; skew 500 ppm
Sep 28 17:49:25 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:25.973805 19645 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:49:25.974267 19645 webserver.cc:460] Webserver started at http://127.3.222.3:44815/ using document root <none> and password file <none>
I20260928 17:49:25.974478 19645 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:25.974521 19645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:25.974638 19645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:49:25.975932 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "814aab6152794923b2acb0df635c3595"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "eafc98435edaf9e0703a9683dad86c76"
server_key_iv: "3f310fd374638ad3d8501dd2d66cf4ad"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.976294 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "814aab6152794923b2acb0df635c3595"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "eafc98435edaf9e0703a9683dad86c76"
server_key_iv: "3f310fd374638ad3d8501dd2d66cf4ad"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.977685 19645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:49:25.978374 19660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.978482 19645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:49:25.978538 19645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "814aab6152794923b2acb0df635c3595"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "eafc98435edaf9e0703a9683dad86c76"
server_key_iv: "3f310fd374638ad3d8501dd2d66cf4ad"
server_key_version: "encryptionkey@0"
I20260928 17:49:25.978591 19645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:25.989588 19645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:25.991374 19645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:25.991498 19645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:25.991676 19645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:25.991958 19645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:25.991991 19645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.992017 19645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:25.992049 19645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:25.997391 19645 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:44669
I20260928 17:49:25.997447 19773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:44669 every 8 connection(s)
I20260928 17:49:25.997709 19645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:49:25.999691 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 19645
I20260928 17:49:25.999786 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:49:25.999951 3960 external_mini_cluster.cc:1428] Setting key c0d6b26974f0d3ca5a10bca9f0f2465c
Sep 28 17:49:26 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:26.006580 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:25.998338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:47575 (local address 127.3.222.62:36989)
0928 17:49:25.998398 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:25.998400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:25.998564 (+ 164us) server_negotiation.cc:407] Connection header received
0928 17:49:25.999007 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:25.999008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:25.999017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:25.999080 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:26.000163 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.000895 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.001823 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.001997 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.002042 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.004274 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:26.004276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:26.004276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:26.004282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:26.005408 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:26.005758 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:26.005759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:26.005760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:26.005818 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:26.006026 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:26.006026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:26.006027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:26.006129 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:26.006181 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:26.006444 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 17:49:26.006517 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 17:49:26.006719 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:25.998297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:47575)
0928 17:49:25.998448 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:25.998487 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:25.998936 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:25.999150 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:25.999154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:25.999844 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:26.000075 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.000083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.000969 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.000970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.001761 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.001763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.002065 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.002066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.002165 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.002649 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:26.002659 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:26.004198 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:26.005472 (+ 1274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:26.005479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:26.005480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:26.005679 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:26.005886 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:26.005887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:26.005887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:26.005977 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:26.006204 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:26.006210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:26.006326 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 17:49:26.006415 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 17:49:26.006529 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260928 17:49:26.007134 19774 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36989
I20260928 17:49:26.007234 19774 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:26.007400 19774 heartbeater.cc:511] Master 127.3.222.62:36989 requested a full tablet report, sending...
I20260928 17:49:26.007774 19315 ts_manager.cc:194] Registered new tserver with Master: 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669)
I20260928 17:49:26.008122 19315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:47575
I20260928 17:49:26.011514 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:49:26.018421 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.012861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.3.222.62:36989)
0928 17:49:26.012930 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:26.012932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:26.012995 (+ 63us) server_negotiation.cc:407] Connection header received
0928 17:49:26.013057 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:26.013058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:26.013067 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:26.013112 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:26.013639 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.014100 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.014783 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.014968 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.015020 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.016161 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:26.016162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:26.016163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:26.016170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:26.017386 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:26.017691 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:26.017691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:26.017692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:26.017744 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:26.017904 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:26.017905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:26.017905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:26.018050 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:26.018137 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:26.018245 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 17:49:26.018325 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 17:49:26.019876 19315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:49:26.021414 19315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:49:26.027621 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.023368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37905 (local address 127.0.0.1:54988)
0928 17:49:26.023708 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:26.023737 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:26.023977 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:26.024207 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:26.024208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:26.024210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:26.024255 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.024256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.025527 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.025528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.026511 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.026513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.027384 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.027384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.027436 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.027439 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:49:26.027488 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 17:49:26.027519 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":58,"threads_started":1}
I20260928 17:49:26.027712 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.023669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.3.222.2:45401)
0928 17:49:26.023833 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:26.023837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:26.023909 (+ 72us) server_negotiation.cc:407] Connection header received
0928 17:49:26.024007 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:26.024010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:26.024026 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:26.024091 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:26.024279 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.025010 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.026375 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.027113 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.027188 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.027269 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:26.027595 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 17:49:26.027637 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260928 17:49:26.027630 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.023394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:44669 (local address 127.0.0.1:52656)
0928 17:49:26.023837 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:26.023846 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:26.023906 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:26.024078 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:26.024081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:26.024083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:26.024147 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.024149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.025475 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.025476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.026536 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.026539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.027390 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.027391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.027454 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.027458 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:49:26.027513 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 17:49:26.027548 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":73,"threads_started":1}
I20260928 17:49:26.027776 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.023491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.3.222.3:44669)
0928 17:49:26.023812 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:26.023816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:26.023904 (+ 88us) server_negotiation.cc:407] Connection header received
0928 17:49:26.023989 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:26.023992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:26.024007 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:26.024233 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:26.024309 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.025375 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.026617 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.027317 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.027376 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.027424 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:26.027665 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 17:49:26.027705 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":73,"threads_started":1}
I20260928 17:49:26.027518 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.023567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:45401 (local address 127.0.0.1:32876)
0928 17:49:26.023855 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:26.023863 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:26.023921 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:26.024170 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:26.024170 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:26.024172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:26.024210 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.024211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.025098 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.025099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.026284 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.026286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.027200 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.027201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.027304 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.027318 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:49:26.027390 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 17:49:26.027443 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":39,"threads_started":1}
I20260928 17:49:26.027632 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.023481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.3.222.1:37905)
0928 17:49:26.023725 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:26.023730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:26.023799 (+ 69us) server_negotiation.cc:407] Connection header received
0928 17:49:26.024045 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:26.024047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:26.024061 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:26.024126 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:26.024332 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.025430 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.026597 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.027192 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.027293 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.027352 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:26.027506 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 17:49:26.027547 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
I20260928 17:49:26.028479 19708 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1f85f5c0ae4ed98df17d04396a3952 (DEFAULT_TABLE table=test-table [id=1ad95cda88324fdb84353b6689272612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:26.028602 19574 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1f85f5c0ae4ed98df17d04396a3952 (DEFAULT_TABLE table=test-table [id=1ad95cda88324fdb84353b6689272612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:26.028839 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1f85f5c0ae4ed98df17d04396a3952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:26.028762 19440 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1f85f5c0ae4ed98df17d04396a3952 (DEFAULT_TABLE table=test-table [id=1ad95cda88324fdb84353b6689272612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:26.028895 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1f85f5c0ae4ed98df17d04396a3952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:26.029050 19440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1f85f5c0ae4ed98df17d04396a3952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:26.031267 19788 tablet_bootstrap.cc:492] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: Bootstrap starting.
I20260928 17:49:26.031843 19788 tablet_bootstrap.cc:654] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:26.031976 19790 tablet_bootstrap.cc:492] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Bootstrap starting.
I20260928 17:49:26.032544 19788 log.cc:824] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:26.032621 19790 tablet_bootstrap.cc:654] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:26.033133 19790 log.cc:824] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:26.033293 19788 tablet_bootstrap.cc:492] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: No bootstrap required, opened a new log
I20260928 17:49:26.033362 19788 ts_tablet_manager.cc:1397] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:49:26.033475 19789 tablet_bootstrap.cc:492] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: Bootstrap starting.
I20260928 17:49:26.033806 19790 tablet_bootstrap.cc:492] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: No bootstrap required, opened a new log
I20260928 17:49:26.033866 19790 ts_tablet_manager.cc:1397] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:49:26.034076 19789 tablet_bootstrap.cc:654] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:26.034595 19789 log.cc:824] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:26.035390 19789 tablet_bootstrap.cc:492] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: No bootstrap required, opened a new log
I20260928 17:49:26.035300 19788 raft_consensus.cc:348] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:26.035439 19788 raft_consensus.cc:374] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:26.035460 19789 ts_tablet_manager.cc:1397] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:49:26.035470 19788 raft_consensus.cc:729] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f523e4cf034c049da74bc97f85cd49, State: Initialized, Role: FOLLOWER
I20260928 17:49:26.035597 19788 consensus_queue.cc:260] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:26.035563 19790 raft_consensus.cc:348] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:26.035717 19790 raft_consensus.cc:374] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:26.035750 19790 raft_consensus.cc:729] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2f00c73482424f9bd055baae51e4e7, State: Initialized, Role: FOLLOWER
I20260928 17:49:26.035856 19790 consensus_queue.cc:260] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:26.036154 19506 heartbeater.cc:503] Master 127.3.222.62:36989 was elected leader, sending a full tablet report...
I20260928 17:49:26.036769 19788 ts_tablet_manager.cc:1428] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 17:49:26.036891 19640 heartbeater.cc:503] Master 127.3.222.62:36989 was elected leader, sending a full tablet report...
I20260928 17:49:26.036943 19790 ts_tablet_manager.cc:1428] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 17:49:26.037565 19789 raft_consensus.cc:348] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:26.037729 19789 raft_consensus.cc:374] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:26.037765 19789 raft_consensus.cc:729] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814aab6152794923b2acb0df635c3595, State: Initialized, Role: FOLLOWER
I20260928 17:49:26.037881 19789 consensus_queue.cc:260] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:26.038269 19789 ts_tablet_manager.cc:1428] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:49:26.038293 19774 heartbeater.cc:503] Master 127.3.222.62:36989 was elected leader, sending a full tablet report...
W20260928 17:49:26.103982 19641 tablet.cc:2369] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:49:26.203248 19507 tablet.cc:2369] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:49:26.248266 19775 tablet.cc:2369] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:49:26.262241 19794 raft_consensus.cc:482] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:49:26.262388 19794 raft_consensus.cc:504] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:26.262811 19794 leader_election.cc:288] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905), 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669)
I20260928 17:49:26.265913 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.263547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:44405 (local address 127.3.222.3:44669)
0928 17:49:26.263666 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:26.263668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:26.263677 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:49:26.263703 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:26.263704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:26.263715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:26.263757 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:26.263876 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.264365 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.265244 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.265611 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.265668 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.265679 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:26.265810 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 17:49:26.265845 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:49:26.265916 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.263091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:51601 (local address 127.3.222.1:37905)
0928 17:49:26.263157 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:26.263159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:26.263336 (+ 177us) server_negotiation.cc:407] Connection header received
0928 17:49:26.263373 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:26.263375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:26.263385 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:26.263441 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:26.263804 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.264278 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.265249 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.265610 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.265669 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.265678 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:26.265811 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 17:49:26.265844 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:49:26.265925 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.262984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:44669 (local address 127.3.222.2:44405)
0928 17:49:26.263026 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:26.263039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:26.263108 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:26.263762 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:26.263763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:26.263764 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:26.263809 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.263810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.264443 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.264444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.265164 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.265166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.265676 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.265677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.265752 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.265756 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:49:26.265799 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:49:26.265851 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 17:49:26.265934 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.262984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37905 (local address 127.3.222.2:51601)
0928 17:49:26.263265 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:26.263274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:26.263326 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:26.263449 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:26.263450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:26.263452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:26.263539 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.263541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.264342 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.264343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.265166 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:26.265168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.265679 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.265680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:26.265752 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.265756 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:49:26.265800 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:49:26.265840 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":54,"thread_start_us":159,"threads_started":1}
I20260928 17:49:26.266253 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1f85f5c0ae4ed98df17d04396a3952" candidate_uuid: "d1f523e4cf034c049da74bc97f85cd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814aab6152794923b2acb0df635c3595" is_pre_election: true
I20260928 17:49:26.266304 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1f85f5c0ae4ed98df17d04396a3952" candidate_uuid: "d1f523e4cf034c049da74bc97f85cd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2f00c73482424f9bd055baae51e4e7" is_pre_election: true
I20260928 17:49:26.266420 19728 raft_consensus.cc:2454] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f523e4cf034c049da74bc97f85cd49 in term 0.
I20260928 17:49:26.266446 19460 raft_consensus.cc:2454] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f523e4cf034c049da74bc97f85cd49 in term 0.
I20260928 17:49:26.266629 19530 leader_election.cc:302] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 814aab6152794923b2acb0df635c3595, ca2f00c73482424f9bd055baae51e4e7, d1f523e4cf034c049da74bc97f85cd49; no voters:
I20260928 17:49:26.266781 19794 raft_consensus.cc:2781] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:49:26.266821 19794 raft_consensus.cc:482] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:49:26.266835 19794 raft_consensus.cc:3037] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:26.267550 19794 raft_consensus.cc:504] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:26.267666 19794 leader_election.cc:288] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 1 election: Requested vote from peers ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905), 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669)
I20260928 17:49:26.267855 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1f85f5c0ae4ed98df17d04396a3952" candidate_uuid: "d1f523e4cf034c049da74bc97f85cd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2f00c73482424f9bd055baae51e4e7"
I20260928 17:49:26.267905 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1f85f5c0ae4ed98df17d04396a3952" candidate_uuid: "d1f523e4cf034c049da74bc97f85cd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814aab6152794923b2acb0df635c3595"
I20260928 17:49:26.267936 19460 raft_consensus.cc:3037] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:26.267974 19728 raft_consensus.cc:3037] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:26.268610 19728 raft_consensus.cc:2454] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f523e4cf034c049da74bc97f85cd49 in term 1.
I20260928 17:49:26.268776 19528 leader_election.cc:302] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 814aab6152794923b2acb0df635c3595, d1f523e4cf034c049da74bc97f85cd49; no voters:
I20260928 17:49:26.268838 19460 raft_consensus.cc:2454] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f523e4cf034c049da74bc97f85cd49 in term 1.
I20260928 17:49:26.268869 19794 raft_consensus.cc:2781] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:49:26.269021 19794 raft_consensus.cc:686] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 1 LEADER]: Becoming Leader. State: Replica: d1f523e4cf034c049da74bc97f85cd49, State: Running, Role: LEADER
I20260928 17:49:26.269111 19794 consensus_queue.cc:237] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:26.270017 19314 catalog_manager.cc:5495] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f523e4cf034c049da74bc97f85cd49 (127.3.222.2). New cstate: current_term: 1 leader_uuid: "d1f523e4cf034c049da74bc97f85cd49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } health_report { overall_health: UNKNOWN } } }
I20260928 17:49:26.311367 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.309276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.3.222.2:45401)
0928 17:49:26.309323 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:26.309325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:26.309353 (+ 28us) server_negotiation.cc:407] Connection header received
0928 17:49:26.309385 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:26.309386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:26.309406 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:26.309458 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:49:26.309569 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.310036 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.310731 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:26.310916 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:26.310963 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:26.311068 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:49:26.311171 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:49:26.311250 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:26.311271 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 17:49:26.311290 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:49:26.313141 19728 raft_consensus.cc:1261] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [term 1 FOLLOWER]: Refusing update from remote peer d1f523e4cf034c049da74bc97f85cd49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:26.313160 19460 raft_consensus.cc:1261] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Refusing update from remote peer d1f523e4cf034c049da74bc97f85cd49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:26.313498 19798 consensus_queue.cc:1035] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:26.313664 19794 consensus_queue.cc:1035] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:26.315754 19801 mvcc.cc:205] Tried to move back new op lower bound from 7334370370816098304 to 7334370370638888960. Current Snapshot: MvccSnapshot[applied={T|T < 7334370370816098304}]
I20260928 17:49:26.316105 19803 mvcc.cc:205] Tried to move back new op lower bound from 7334370370816098304 to 7334370370638888960. Current Snapshot: MvccSnapshot[applied={T|T < 7334370370816098304}]
I20260928 17:49:26.318114 19315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:26.318210 19315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:26.318877 19315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Sending DeleteTablet for 3 replicas of tablet 7a1f85f5c0ae4ed98df17d04396a3952
I20260928 17:49:26.319190 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 19377
I20260928 17:49:26.319171 19708 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1f85f5c0ae4ed98df17d04396a3952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:26 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:52656
I20260928 17:49:26.319185 19440 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1f85f5c0ae4ed98df17d04396a3952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:26 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:54988
I20260928 17:49:26.319857 19574 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1f85f5c0ae4ed98df17d04396a3952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:26 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:32876
I20260928 17:49:26.320122 19810 tablet_replica.cc:330] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: stopping tablet replica
I20260928 17:49:26.320221 19810 raft_consensus.cc:2229] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:49:26.320317 19810 raft_consensus.cc:2258] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:26.320461 19811 tablet_replica.cc:330] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: stopping tablet replica
I20260928 17:49:26.320546 19811 raft_consensus.cc:2229] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:49:26.320659 19811 raft_consensus.cc:2258] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:26.320884 19810 ts_tablet_manager.cc:1905] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:26.321065 19811 ts_tablet_manager.cc:1905] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:26.323087 19811 ts_tablet_manager.cc:1918] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:49:26.323118 19810 ts_tablet_manager.cc:1918] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:49:26.323175 19810 log.cc:1196] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/7a1f85f5c0ae4ed98df17d04396a3952
I20260928 17:49:26.323297 19811 log.cc:1196] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/7a1f85f5c0ae4ed98df17d04396a3952
I20260928 17:49:26.323506 19810 ts_tablet_manager.cc:1939] T 7a1f85f5c0ae4ed98df17d04396a3952 P 814aab6152794923b2acb0df635c3595: Deleting consensus metadata
I20260928 17:49:26.323676 19811 ts_tablet_manager.cc:1939] T 7a1f85f5c0ae4ed98df17d04396a3952 P d1f523e4cf034c049da74bc97f85cd49: Deleting consensus metadata
I20260928 17:49:26.323884 19300 catalog_manager.cc:4841] TS 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669): tablet 7a1f85f5c0ae4ed98df17d04396a3952 (table test-table [id=1ad95cda88324fdb84353b6689272612]) successfully deleted
I20260928 17:49:26.324087 19302 catalog_manager.cc:4841] TS d1f523e4cf034c049da74bc97f85cd49 (127.3.222.2:45401): tablet 7a1f85f5c0ae4ed98df17d04396a3952 (table test-table [id=1ad95cda88324fdb84353b6689272612]) successfully deleted
W20260928 17:49:26.325963 19302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:37905 (error 108)
I20260928 17:49:26.326413 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:26.326296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37905 (local address 127.0.0.1:54996)
0928 17:49:26.326345 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:26.326380 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:37905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 17:49:26.326511 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 19511
W20260928 17:49:26.327826 19302 catalog_manager.cc:4584] TS ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a1f85f5c0ae4ed98df17d04396a3952: Network error: Client connection negotiation failed: client connection to 127.3.222.1:37905: connect: Connection refused (error 111)
I20260928 17:49:26.330664 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 19645
I20260928 17:49:26.334456 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 19283
I20260928 17:49:46.338900 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36989
--webserver_interface=127.3.222.62
--webserver_port=36263
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:36989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:49:46.416909 19813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:46.417093 19813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:46.417112 19813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:46.418170 19813 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 17:49:46.418202 19813 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 17:49:46.418213 19813 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 17:49:46.418223 19813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:49:46.418234 19813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:46.418246 19813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:46.418256 19813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:49:46.418272 19813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:49:46.419701 19813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:36989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=36263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:46.419884 19813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:46.420039 19813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:46.427747 19822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:46.427870 19819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:46.427937 19813 server_base.cc:1034] running on GCE node
W20260928 17:49:46.428026 19820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:46.428262 19813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:46.428514 19813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:46.429658 19813 hybrid_clock.cc:648] HybridClock initialized: now 1790617786429646 us; error 33 us; skew 500 ppm
Sep 28 17:49:46 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:46.431991 19813 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:49:46.432384 19813 webserver.cc:460] Webserver started at http://127.3.222.62:36263/ using document root <none> and password file <none>
I20260928 17:49:46.432610 19813 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:46.432663 19813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:46.433988 19813 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:49:46.434599 19829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:46.434760 19813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:49:46.434846 19813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "e90fc26731db4c4bb93d09cd61f522ca"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "efc4c0c80149a11af7d1526b771886a9"
server_key_iv: "6b61babd9e4ffa7cd0576d6e48a1f90f"
server_key_version: "encryptionkey@0"
I20260928 17:49:46.435204 19813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:46.443005 19813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:46.444546 19813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:46.444664 19813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:46.448246 19813 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:36989
I20260928 17:49:46.448284 19881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:36989 every 8 connection(s)
I20260928 17:49:46.448607 19813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:49:46.449925 19882 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:49:46.450551 19882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Bootstrap starting.
I20260928 17:49:46.453404 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 19813
I20260928 17:49:46.453734 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:37905
--local_ip_for_outbound_sockets=127.3.222.1
--tserver_master_addrs=127.3.222.62:36989
--webserver_port=43367
--webserver_interface=127.3.222.1
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:46.458935 19882 log.cc:824] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:46.460669 19882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:49:46.460916 19882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Bootstrap complete.
I20260928 17:49:46.462890 19882 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:49:46.463102 19882 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90fc26731db4c4bb93d09cd61f522ca, State: Initialized, Role: FOLLOWER
I20260928 17:49:46.463271 19882 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:49:46.463335 19882 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:49:46.463384 19882 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:49:46.463438 19882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:49:46.464367 19882 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:49:46.464464 19882 leader_election.cc:302] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90fc26731db4c4bb93d09cd61f522ca; no voters:
I20260928 17:49:46.464610 19882 leader_election.cc:288] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 17:49:46.464653 19886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:49:46.464761 19886 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 2 LEADER]: Becoming Leader. State: Replica: e90fc26731db4c4bb93d09cd61f522ca, State: Running, Role: LEADER
I20260928 17:49:46.464862 19882 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:49:46.464839 19886 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:49:46.465111 19887 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e90fc26731db4c4bb93d09cd61f522ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } } }
I20260928 17:49:46.465200 19887 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:46.465359 19888 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90fc26731db4c4bb93d09cd61f522ca. Latest consensus state: current_term: 2 leader_uuid: "e90fc26731db4c4bb93d09cd61f522ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } } }
I20260928 17:49:46.465415 19888 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: This master's current role is: LEADER
I20260928 17:49:46.465503 19892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:49:46.466226 19892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:49:46.466318 19892 catalog_manager.cc:1259] Loaded cluster ID: cfc818a997094d9d9cd7297d97192294
I20260928 17:49:46.466543 19892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:49:46.467545 19892 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:49:46.467712 19892 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Loaded TSK: 0
I20260928 17:49:46.475904 19892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Generated new TSK 1
I20260928 17:49:46.475986 19892 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 17:49:46.552785 19884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:46.552974 19884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:46.552996 19884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:46.554102 19884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:46.554140 19884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:46.554164 19884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:49:46.555593 19884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:37905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=43367
--tserver_master_addrs=127.3.222.62:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:46.555799 19884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:46.555999 19884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:46.560711 19910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:46.560727 19909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:46.560711 19912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:46.561029 19884 server_base.cc:1034] running on GCE node
I20260928 17:49:46.561210 19884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:46.561419 19884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:46.562564 19884 hybrid_clock.cc:648] HybridClock initialized: now 1790617786562551 us; error 30 us; skew 500 ppm
Sep 28 17:49:46 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:46.564925 19884 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:49:46.565326 19884 webserver.cc:460] Webserver started at http://127.3.222.1:43367/ using document root <none> and password file <none>
I20260928 17:49:46.565547 19884 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:46.565606 19884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:46.566953 19884 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:49:46.567584 19919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:46.567728 19884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:46.567807 19884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "ca2f00c73482424f9bd055baae51e4e7"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "1ebd54727164029558fb55bdf78e7d49"
server_key_iv: "ca8711ec4f01e757dc4ba20dfb639e72"
server_key_version: "encryptionkey@0"
I20260928 17:49:46.568135 19884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:46.577622 19884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:46.579581 19884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:46.579694 19884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:46.579882 19884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:46.580288 19926 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:49:46.581612 19884 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:49:46.581656 19884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260928 17:49:46.581677 19884 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:49:46.582362 19884 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:49:46.582409 19884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 17:49:46.582468 19926 tablet_bootstrap.cc:492] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Bootstrap starting.
I20260928 17:49:46.589430 19884 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:37905
I20260928 17:49:46.589569 20033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:37905 every 8 connection(s)
I20260928 17:49:46.589887 19884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:49:46.593289 19926 log.cc:824] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:46.595471 19926 tablet_bootstrap.cc:492] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:49:46.595753 19926 tablet_bootstrap.cc:492] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Bootstrap complete.
I20260928 17:49:46.596653 19926 ts_tablet_manager.cc:1397] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260928 17:49:46.597628 19926 raft_consensus.cc:348] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
I20260928 17:49:46.597923 19926 raft_consensus.cc:729] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2f00c73482424f9bd055baae51e4e7, State: Initialized, Role: FOLLOWER
I20260928 17:49:46.598080 19926 consensus_queue.cc:260] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } } peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } }
Sep 28 17:49:46 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:46.598361 19926 ts_tablet_manager.cc:1428] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:49:46.598728 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 19884
I20260928 17:49:46.599011 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:45401
--local_ip_for_outbound_sockets=127.3.222.2
--tserver_master_addrs=127.3.222.62:36989
--webserver_port=42493
--webserver_interface=127.3.222.2
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:46.602005 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.590535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:60185)
0928 17:49:46.590706 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.590752 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:46.591428 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:46.591904 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:46.591909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:46.592286 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:46.592603 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.592610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.594302 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.594303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.595521 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.595524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.596306 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.596308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.596456 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.597033 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:46.597042 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:46.598649 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:46.600448 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:46.600454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:46.600456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:46.600721 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:46.600976 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:46.600976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:46.600976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:46.601086 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:46.601414 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:46.601418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:46.601556 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 17:49:46.601657 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 17:49:46.601805 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260928 17:49:46.602033 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.590632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:60185 (local address 127.3.222.62:36989)
0928 17:49:46.590832 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.590840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.590881 (+ 41us) server_negotiation.cc:407] Connection header received
0928 17:49:46.591648 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.591655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.591711 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.591849 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:46.592749 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.594178 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.595614 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.596193 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.596291 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.598744 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:46.598747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:46.598747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:46.598756 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:46.600349 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.600794 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.600797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.600798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.600914 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.601142 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.601142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.601142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.601359 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:46.601424 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.601726 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.601851 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260928 17:49:46.602854 20034 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36989
I20260928 17:49:46.602952 20034 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:46.603111 20034 heartbeater.cc:511] Master 127.3.222.62:36989 requested a full tablet report, sending...
I20260928 17:49:46.603904 19846 ts_manager.cc:194] Registered new tserver with Master: ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905)
I20260928 17:49:46.605050 19846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:60185
I20260928 17:49:46.605993 20034 heartbeater.cc:503] Master 127.3.222.62:36989 was elected leader, sending a full tablet report...
Sep 28 17:49:46 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
I20260928 17:49:46.614162 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.604715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.3.222.1:37905)
0928 17:49:46.604946 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.604949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.604962 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:49:46.604990 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.604992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.605004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.606408 (+ 1404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:46.606499 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.607564 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.608673 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.609107 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.609443 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.611350 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:46.611357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:46.611358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:46.611364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:46.613013 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.613393 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.613403 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.613403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.613461 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.613621 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.613622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.613622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.613757 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:46.613828 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.613990 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.614068 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":67,"threads_started":1}
I20260928 17:49:46.614166 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.604502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37905 (local address 127.0.0.1:47972)
0928 17:49:46.604743 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.604774 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:46.604828 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:46.605082 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:46.605085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:46.606275 (+ 1190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:46.606331 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.606333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.607655 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.607656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.608571 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.608574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.609195 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.609196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.609269 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.609725 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:46.609730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:46.611264 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:46.613084 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:46.613090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:46.613092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:46.613323 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:46.613517 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:46.613518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:46.613518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:46.613573 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:46.613812 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:46.613817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:46.613865 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 17:49:46.613956 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 17:49:46.614068 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":69,"threads_started":1}
I20260928 17:49:46.614902 19968 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1f85f5c0ae4ed98df17d04396a3952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:26 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:47972
I20260928 17:49:46.615486 20043 tablet_replica.cc:330] stopping tablet replica
I20260928 17:49:46.615624 20043 raft_consensus.cc:2229] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:49:46.615721 20043 raft_consensus.cc:2258] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:46.616079 20043 ts_tablet_manager.cc:1905] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:46.618851 20043 ts_tablet_manager.cc:1918] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:49:46.618959 20043 log.cc:1196] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal/wals/7a1f85f5c0ae4ed98df17d04396a3952
I20260928 17:49:46.619144 20043 ts_tablet_manager.cc:1939] T 7a1f85f5c0ae4ed98df17d04396a3952 P ca2f00c73482424f9bd055baae51e4e7: Deleting consensus metadata
I20260928 17:49:46.619601 19833 catalog_manager.cc:4841] TS ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905): tablet 7a1f85f5c0ae4ed98df17d04396a3952 (table test-table [id=1ad95cda88324fdb84353b6689272612]) successfully deleted
W20260928 17:49:46.680789 20040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:46.680962 20040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:46.680982 20040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:46.682029 20040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:46.682058 20040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:46.682082 20040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:49:46.683516 20040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:45401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=42493
--tserver_master_addrs=127.3.222.62:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:46.683679 20040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:46.683844 20040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:46.688697 20051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:46.688741 20040 server_base.cc:1034] running on GCE node
W20260928 17:49:46.688710 20048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:46.688710 20049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:46.689095 20040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:46.689282 20040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:46.690459 20040 hybrid_clock.cc:648] HybridClock initialized: now 1790617786690444 us; error 25 us; skew 500 ppm
Sep 28 17:49:46 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:46.692848 20040 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:49:46.693219 20040 webserver.cc:460] Webserver started at http://127.3.222.2:42493/ using document root <none> and password file <none>
I20260928 17:49:46.693414 20040 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:46.693477 20040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:46.694700 20040 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:49:46.695309 20058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:46.695472 20040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:49:46.695539 20040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "d1f523e4cf034c049da74bc97f85cd49"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "f2531a55e2cf1648ea436f509588ccc0"
server_key_iv: "1da764b56db9ec65b8cc935c93c9767a"
server_key_version: "encryptionkey@0"
I20260928 17:49:46.695863 20040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:46.708097 20040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:46.709848 20040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:46.709966 20040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:46.710162 20040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:46.710427 20040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:46.710458 20040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:46.710479 20040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:46.710502 20040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:46.715977 20040 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:45401
I20260928 17:49:46.716012 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:45401 every 8 connection(s)
I20260928 17:49:46.716310 20040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:49:46 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:46.724462 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.716922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:55655 (local address 127.3.222.62:36989)
0928 17:49:46.716963 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.716965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.717154 (+ 189us) server_negotiation.cc:407] Connection header received
0928 17:49:46.717680 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.717681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.717697 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.717749 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:46.718639 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.719128 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.720081 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.720274 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.720327 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.722197 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:46.722198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:46.722198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:46.722206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:46.723271 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.723626 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.723627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.723628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.723678 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.723884 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.723885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.723886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.723990 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:46.724043 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.724287 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.724373 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:49:46.724606 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.716906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:55655)
0928 17:49:46.717052 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.717084 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:46.717587 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:46.717824 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:46.717828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:46.718370 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:46.718548 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.718553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.719228 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.719230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.719999 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.720001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.720341 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.720342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.720458 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.720890 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:46.720898 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:46.722114 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:46.723339 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:46.723345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:46.723347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:46.723548 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:46.723734 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:46.723735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:46.723735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:46.723830 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:46.724041 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:46.724047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:46.724171 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 17:49:46.724253 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 17:49:46.724407 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260928 17:49:46.724709 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 20040
I20260928 17:49:46.725009 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:44669
--local_ip_for_outbound_sockets=127.3.222.3
--tserver_master_addrs=127.3.222.62:36989
--webserver_port=44815
--webserver_interface=127.3.222.3
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:49:46.725031 20172 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36989
I20260928 17:49:46.725121 20172 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:46.725277 20172 heartbeater.cc:511] Master 127.3.222.62:36989 requested a full tablet report, sending...
I20260928 17:49:46.725632 19846 ts_manager.cc:194] Registered new tserver with Master: d1f523e4cf034c049da74bc97f85cd49 (127.3.222.2:45401)
I20260928 17:49:46.726066 19846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:55655
W20260928 17:49:46.802552 20175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:49:46.802762 20175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:49:46.802793 20175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:49:46.803911 20175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:49:46.803948 20175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:49:46.803982 20175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:49:46.805330 20175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:44669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=44815
--tserver_master_addrs=127.3.222.62:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:49:46.805540 20175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:49:46.805735 20175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:49:46.810292 20180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:46.810302 20183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:49:46.810302 20181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:49:46.810537 20175 server_base.cc:1034] running on GCE node
I20260928 17:49:46.810708 20175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:49:46.810927 20175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:49:46.812076 20175 hybrid_clock.cc:648] HybridClock initialized: now 1790617786812063 us; error 30 us; skew 500 ppm
Sep 28 17:49:46 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:49:46.814424 20175 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:49:46.814975 20175 webserver.cc:460] Webserver started at http://127.3.222.3:44815/ using document root <none> and password file <none>
I20260928 17:49:46.815178 20175 fs_manager.cc:359] Metadata directory not provided
I20260928 17:49:46.815253 20175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:49:46.816573 20175 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:49:46.817176 20190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:46.817342 20175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:49:46.817420 20175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "814aab6152794923b2acb0df635c3595"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "eafc98435edaf9e0703a9683dad86c76"
server_key_iv: "3f310fd374638ad3d8501dd2d66cf4ad"
server_key_version: "encryptionkey@0"
I20260928 17:49:46.817781 20175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:49:46.833087 20175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:49:46.834738 20175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:49:46.834860 20175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:49:46.835049 20175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:49:46.835357 20175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:49:46.835390 20175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:46.835418 20175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:49:46.835441 20175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:49:46.841014 20175 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:44669
I20260928 17:49:46.841073 20303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:44669 every 8 connection(s)
I20260928 17:49:46.841393 20175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:49:46 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:49:46.849783 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 20175
I20260928 17:49:46.849937 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.842062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:55257 (local address 127.3.222.62:36989)
0928 17:49:46.842126 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.842128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.842342 (+ 214us) server_negotiation.cc:407] Connection header received
0928 17:49:46.842786 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.842787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.842798 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.842857 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:46.843789 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.844277 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.845256 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.845447 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.845505 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.847550 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:46.847551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:46.847552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:46.847558 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:46.848811 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.849139 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.849140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.849140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.849186 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.849358 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.849359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.849359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.849451 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:46.849494 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.849780 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.849850 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:49:46.850162 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.842047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:55257)
0928 17:49:46.842196 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.842237 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:46.842702 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:46.842934 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:46.842938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:46.843517 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:49:46.843703 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.843708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.844348 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.844349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.845169 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.845172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.845510 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.845510 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.845605 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.846044 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:49:46.846053 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:49:46.847446 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:49:46.848883 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:46.848889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:46.848891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:46.849085 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:46.849231 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:49:46.849232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:49:46.849232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:49:46.849317 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:49:46.849500 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:49:46.849504 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:49:46.849626 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 17:49:46.849732 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 17:49:46.849897 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260928 17:49:46.850576 20304 heartbeater.cc:347] Connected to a master server at 127.3.222.62:36989
I20260928 17:49:46.850665 20304 heartbeater.cc:464] Registering TS with master...
I20260928 17:49:46.850837 20304 heartbeater.cc:511] Master 127.3.222.62:36989 requested a full tablet report, sending...
I20260928 17:49:46.851267 19846 ts_manager.cc:194] Registered new tserver with Master: 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669)
I20260928 17:49:46.851624 19846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:55257
I20260928 17:49:46.856312 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.850203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.3.222.62:36989)
0928 17:49:46.850255 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.850256 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.850357 (+ 101us) server_negotiation.cc:407] Connection header received
0928 17:49:46.850449 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.850449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.850459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.850506 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:49:46.851053 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.851717 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.852684 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.852892 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.852944 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.854005 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:49:46.854006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:49:46.854006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:49:46.854013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:49:46.855084 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.855523 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.855524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.855525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.855581 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:49:46.855754 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:49:46.855755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:49:46.855755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:49:46.855879 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:49:46.856045 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.856080 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.856146 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:49:46.857028 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:49:46.859617 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.857413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.3.222.62:36989)
0928 17:49:46.857461 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.857462 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.857560 (+ 98us) server_negotiation.cc:407] Connection header received
0928 17:49:46.857616 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.857616 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.857644 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.857689 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:49:46.857765 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.858191 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.858966 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.859146 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.859193 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.859344 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:49:46.859430 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:49:46.859487 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.859556 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.859569 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:49:46.859887 19846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:49:46.861413 19846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:49:46.863513 19968 tablet_service.cc:1467] Processing CreateTablet for tablet d1e2cfa3d8a6438aae11c70657051495 (DEFAULT_TABLE table=test-table [id=29ca9622be294cebafbb32543382a53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:46.863705 19968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e2cfa3d8a6438aae11c70657051495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:46.865312 19926 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Bootstrap starting.
I20260928 17:49:46.865641 19926 tablet_bootstrap.cc:654] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:46.866379 19926 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: No bootstrap required, opened a new log
I20260928 17:49:46.866423 19926 ts_tablet_manager.cc:1397] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 17:49:46.866736 19926 raft_consensus.cc:348] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:49:46.866813 19926 raft_consensus.cc:374] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:46.866852 19926 raft_consensus.cc:729] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2f00c73482424f9bd055baae51e4e7, State: Initialized, Role: FOLLOWER
I20260928 17:49:46.866896 19926 consensus_queue.cc:260] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:49:46.866906 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.863058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:44669 (local address 127.0.0.1:57990)
0928 17:49:46.863151 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.863160 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:46.863197 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:46.863578 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:46.863579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:46.863581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:46.863621 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.863623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.864628 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.864629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.865713 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.865716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.866652 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.866653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.866715 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.866734 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 17:49:46.866783 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 17:49:46.866822 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260928 17:49:46.866981 19926 ts_tablet_manager.cc:1428] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 17:49:46.867179 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.863274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.3.222.3:44669)
0928 17:49:46.863437 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.863441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.863450 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:49:46.863477 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.863479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.863491 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.863562 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:46.863694 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.864540 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.865804 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.866493 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.866560 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.866618 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.867060 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.867100 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260928 17:49:46.867226 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.863031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:45401 (local address 127.0.0.1:36874)
0928 17:49:46.863129 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.863139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:46.863172 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:46.863654 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:46.863655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:46.863656 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:46.863683 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.863684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.864887 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.864888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.865902 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.865905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.867004 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.867005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.867067 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.867071 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:49:46.867102 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 17:49:46.867151 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 17:49:46.867717 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.863148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.3.222.2:45401)
0928 17:49:46.863484 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.863488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.863499 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:49:46.863535 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.863537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.863551 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.863654 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:46.863738 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.864790 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.866205 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.866910 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.867512 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.867575 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.867601 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.867642 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":105,"threads_started":1}
I20260928 17:49:46.867911 20238 tablet_service.cc:1467] Processing CreateTablet for tablet d1e2cfa3d8a6438aae11c70657051495 (DEFAULT_TABLE table=test-table [id=29ca9622be294cebafbb32543382a53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:46.868227 20238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e2cfa3d8a6438aae11c70657051495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:46.868316 20106 tablet_service.cc:1467] Processing CreateTablet for tablet d1e2cfa3d8a6438aae11c70657051495 (DEFAULT_TABLE table=test-table [id=29ca9622be294cebafbb32543382a53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:49:46.868587 20106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e2cfa3d8a6438aae11c70657051495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:49:46.870846 20313 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Bootstrap starting.
I20260928 17:49:46.871227 20314 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Bootstrap starting.
I20260928 17:49:46.871544 20313 tablet_bootstrap.cc:654] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:46.871766 20314 tablet_bootstrap.cc:654] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Neither blocks nor log segments found. Creating new log.
I20260928 17:49:46.872038 20313 log.cc:824] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:46.872330 20314 log.cc:824] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Log is configured to *not* fsync() on all Append() calls
I20260928 17:49:46.872776 20313 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: No bootstrap required, opened a new log
I20260928 17:49:46.872833 20313 ts_tablet_manager.cc:1397] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:49:46.872982 20314 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: No bootstrap required, opened a new log
I20260928 17:49:46.873037 20314 ts_tablet_manager.cc:1397] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:49:46.874117 20313 raft_consensus.cc:348] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:49:46.874257 20313 raft_consensus.cc:374] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:46.874281 20313 raft_consensus.cc:729] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814aab6152794923b2acb0df635c3595, State: Initialized, Role: FOLLOWER
I20260928 17:49:46.874377 20313 consensus_queue.cc:260] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:49:46.874286 20314 raft_consensus.cc:348] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:49:46.874451 20314 raft_consensus.cc:374] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:49:46.874492 20314 raft_consensus.cc:729] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f523e4cf034c049da74bc97f85cd49, State: Initialized, Role: FOLLOWER
I20260928 17:49:46.874626 20304 heartbeater.cc:503] Master 127.3.222.62:36989 was elected leader, sending a full tablet report...
I20260928 17:49:46.874600 20314 consensus_queue.cc:260] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:49:46.874667 20313 ts_tablet_manager.cc:1428] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:49:46.874857 20314 ts_tablet_manager.cc:1428] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:49:46.874890 20172 heartbeater.cc:503] Master 127.3.222.62:36989 was elected leader, sending a full tablet report...
I20260928 17:49:46.929834 20317 raft_consensus.cc:482] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:49:46.929961 20317 raft_consensus.cc:504] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:49:46.930295 20317 leader_election.cc:288] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f523e4cf034c049da74bc97f85cd49 (127.3.222.2:45401), ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905)
I20260928 17:49:46.933992 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.930611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:44021 (local address 127.3.222.2:45401)
0928 17:49:46.930670 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.930672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.930681 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:49:46.930704 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.930704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.930714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.930757 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:46.930938 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.931695 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.933082 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.933639 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.933709 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.933721 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.933902 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.933939 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:49:46.933982 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.930628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:37927 (local address 127.3.222.1:37905)
0928 17:49:46.930671 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.930673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.930786 (+ 113us) server_negotiation.cc:407] Connection header received
0928 17:49:46.930827 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.930828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.930853 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.930924 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:49:46.931045 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.931748 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.933082 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.933674 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.933730 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.933800 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.933869 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.933913 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:49:46.933983 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.930434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:45401 (local address 127.3.222.3:44021)
0928 17:49:46.930479 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.930489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:46.930531 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:46.930812 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:46.930812 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:46.930814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:46.930876 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.930877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.931790 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.931792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.933003 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.933006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.933714 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.933716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.933843 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.933845 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:49:46.933891 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:49:46.933919 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":41}
I20260928 17:49:46.933995 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.930538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37905 (local address 127.3.222.3:37927)
0928 17:49:46.930719 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.930727 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:49:46.930781 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:49:46.930938 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:49:46.930939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:49:46.930941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:49:46.930991 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.930992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.931810 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.931810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.933003 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:49:46.933006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.933734 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.933735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:49:46.933798 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.933803 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:49:46.933856 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 17:49:46.933896 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":17,"thread_start_us":109,"threads_started":1}
I20260928 17:49:46.934315 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e2cfa3d8a6438aae11c70657051495" candidate_uuid: "814aab6152794923b2acb0df635c3595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2f00c73482424f9bd055baae51e4e7" is_pre_election: true
I20260928 17:49:46.934345 20126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e2cfa3d8a6438aae11c70657051495" candidate_uuid: "814aab6152794923b2acb0df635c3595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f523e4cf034c049da74bc97f85cd49" is_pre_election: true
I20260928 17:49:46.934470 20126 raft_consensus.cc:2454] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 814aab6152794923b2acb0df635c3595 in term 0.
I20260928 17:49:46.934470 19988 raft_consensus.cc:2454] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 814aab6152794923b2acb0df635c3595 in term 0.
I20260928 17:49:46.934676 20194 leader_election.cc:302] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 814aab6152794923b2acb0df635c3595, ca2f00c73482424f9bd055baae51e4e7; no voters:
I20260928 17:49:46.934787 20317 raft_consensus.cc:2781] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:49:46.934839 20317 raft_consensus.cc:482] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:49:46.934861 20317 raft_consensus.cc:3037] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:46.935508 20317 raft_consensus.cc:504] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:49:46.935606 20317 leader_election.cc:288] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [CANDIDATE]: Term 1 election: Requested vote from peers d1f523e4cf034c049da74bc97f85cd49 (127.3.222.2:45401), ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905)
I20260928 17:49:46.935760 20126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e2cfa3d8a6438aae11c70657051495" candidate_uuid: "814aab6152794923b2acb0df635c3595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f523e4cf034c049da74bc97f85cd49"
I20260928 17:49:46.935802 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e2cfa3d8a6438aae11c70657051495" candidate_uuid: "814aab6152794923b2acb0df635c3595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2f00c73482424f9bd055baae51e4e7"
I20260928 17:49:46.935842 20126 raft_consensus.cc:3037] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:46.935873 19988 raft_consensus.cc:3037] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:49:46.936661 20126 raft_consensus.cc:2454] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 814aab6152794923b2acb0df635c3595 in term 1.
I20260928 17:49:46.936661 19988 raft_consensus.cc:2454] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 814aab6152794923b2acb0df635c3595 in term 1.
I20260928 17:49:46.936820 20194 leader_election.cc:302] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 814aab6152794923b2acb0df635c3595, ca2f00c73482424f9bd055baae51e4e7; no voters:
I20260928 17:49:46.936901 20317 raft_consensus.cc:2781] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:49:46.937050 20317 raft_consensus.cc:686] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 1 LEADER]: Becoming Leader. State: Replica: 814aab6152794923b2acb0df635c3595, State: Running, Role: LEADER
I20260928 17:49:46.937142 20317 consensus_queue.cc:237] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:49:46.937887 19845 catalog_manager.cc:5495] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 reported cstate change: term changed from 0 to 1, leader changed from <none> to 814aab6152794923b2acb0df635c3595 (127.3.222.3). New cstate: current_term: 1 leader_uuid: "814aab6152794923b2acb0df635c3595" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } health_report { overall_health: UNKNOWN } } }
I20260928 17:49:46.961136 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.959034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.3.222.3:44669)
0928 17:49:46.959097 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:49:46.959098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:49:46.959108 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:49:46.959138 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:49:46.959139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:49:46.959155 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:49:46.959315 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:49:46.959419 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.959891 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.960582 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:49:46.960767 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:49:46.960820 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:49:46.960915 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:49:46.961005 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:49:46.961057 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:49:46.961069 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 17:49:46.961084 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:49:46.962669 19988 raft_consensus.cc:1261] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Refusing update from remote peer 814aab6152794923b2acb0df635c3595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:46.962682 20126 raft_consensus.cc:1261] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Refusing update from remote peer 814aab6152794923b2acb0df635c3595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:49:46.962951 20317 consensus_queue.cc:1035] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:46.963083 20320 consensus_queue.cc:1035] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:49:46.964877 20323 mvcc.cc:205] Tried to move back new op lower bound from 7334370455396933632 to 7334370455295168512. Current Snapshot: MvccSnapshot[applied={T|T < 7334370455396933632}]
I20260928 17:49:46.965080 20324 mvcc.cc:205] Tried to move back new op lower bound from 7334370455396933632 to 7334370455295168512. Current Snapshot: MvccSnapshot[applied={T|T < 7334370455396933632}]
I20260928 17:49:46.966303 20325 mvcc.cc:205] Tried to move back new op lower bound from 7334370455396933632 to 7334370455295168512. Current Snapshot: MvccSnapshot[applied={T|T < 7334370455396933632 or (T in {7334370455396933632})}]
I20260928 17:49:46.967118 19845 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:46.967226 19845 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:49:46.967857 19845 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Sending DeleteTablet for 3 replicas of tablet d1e2cfa3d8a6438aae11c70657051495
I20260928 17:49:46.968127 20238 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e2cfa3d8a6438aae11c70657051495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:46 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:57990
I20260928 17:49:46.968230 20106 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e2cfa3d8a6438aae11c70657051495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:46 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:36874
I20260928 17:49:46.968256 19968 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e2cfa3d8a6438aae11c70657051495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:49:46 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:47972
I20260928 17:49:46.968348 20043 tablet_replica.cc:330] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: stopping tablet replica
I20260928 17:49:46.968403 20043 raft_consensus.cc:2229] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:49:46.968401 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 19884
I20260928 17:49:46.968456 20043 raft_consensus.cc:2258] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:46.969123 20332 tablet_replica.cc:330] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: stopping tablet replica
I20260928 17:49:46.969216 20332 raft_consensus.cc:2229] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:49:46.969297 20332 raft_consensus.cc:2258] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:49:46.969656 20332 ts_tablet_manager.cc:1905] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:49:46.971395 20332 ts_tablet_manager.cc:1918] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:49:46.971458 20332 log.cc:1196] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/d1e2cfa3d8a6438aae11c70657051495
I20260928 17:49:46.971691 20332 ts_tablet_manager.cc:1939] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Deleting consensus metadata
I20260928 17:49:46.972046 19831 catalog_manager.cc:4841] TS 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669): tablet d1e2cfa3d8a6438aae11c70657051495 (table test-table [id=29ca9622be294cebafbb32543382a53c]) successfully deleted
W20260928 17:49:46.974153 19833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:37905 (error 108)
I20260928 17:49:46.974388 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 20040
I20260928 17:49:46.975179 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.974538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37905 (local address 127.0.0.1:47988)
0928 17:49:46.975105 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.975147 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:37905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548}
W20260928 17:49:46.975929 19833 catalog_manager.cc:4584] TS ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1e2cfa3d8a6438aae11c70657051495: Network error: Client connection negotiation failed: client connection to 127.3.222.1:37905: connect: Connection refused (error 111)
I20260928 17:49:46.981829 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 20175
I20260928 17:49:46.981901 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:49:46.981799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:45401 (local address 127.0.0.1:36884)
0928 17:49:46.981847 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:49:46.981872 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.2:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 17:49:46.985478 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 19813
I20260928 17:50:06.990072 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36989
--webserver_interface=127.3.222.62
--webserver_port=36263
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:36989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:07.068549 20335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:07.068756 20335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:07.068786 20335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:07.070156 20335 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 17:50:07.070214 20335 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 17:50:07.070235 20335 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 17:50:07.070251 20335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:07.070266 20335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:07.070283 20335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:07.070300 20335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:07.070324 20335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:07.071918 20335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:36989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:36989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=36263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:07.072158 20335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:07.072359 20335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:07.077064 20343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:07.077085 20341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:07.077231 20340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:07.077291 20335 server_base.cc:1034] running on GCE node
I20260928 17:50:07.077538 20335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:07.077785 20335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:07.078943 20335 hybrid_clock.cc:648] HybridClock initialized: now 1790617807078928 us; error 29 us; skew 500 ppm
Sep 28 17:50:07 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:07.081456 20335 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:07.081873 20335 webserver.cc:460] Webserver started at http://127.3.222.62:36263/ using document root <none> and password file <none>
I20260928 17:50:07.082062 20335 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:07.082110 20335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:07.083393 20335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:07.083981 20350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:07.084131 20335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:07.084221 20335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "e90fc26731db4c4bb93d09cd61f522ca"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "efc4c0c80149a11af7d1526b771886a9"
server_key_iv: "6b61babd9e4ffa7cd0576d6e48a1f90f"
server_key_version: "encryptionkey@0"
I20260928 17:50:07.084666 20335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:07.097718 20335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:07.099746 20335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:07.099856 20335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:07.103564 20335 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:36989
I20260928 17:50:07.103623 20402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:36989 every 8 connection(s)
I20260928 17:50:07.104022 20335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:07.104560 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 20335
I20260928 17:50:07.104864 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:37905
--local_ip_for_outbound_sockets=127.3.222.1
--tserver_master_addrs=127.3.222.62:36989
--webserver_port=43367
--webserver_interface=127.3.222.1
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:50:07.105875 20403 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:50:07.106626 20403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Bootstrap starting.
I20260928 17:50:07.115862 20403 log.cc:824] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:07.118201 20403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:50:07.118479 20403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Bootstrap complete.
I20260928 17:50:07.120868 20403 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:50:07.121090 20403 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90fc26731db4c4bb93d09cd61f522ca, State: Initialized, Role: FOLLOWER
I20260928 17:50:07.121232 20403 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:50:07.121292 20403 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:07.121320 20403 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:07.121372 20403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 2 FOLLOWER]: Advancing to term 3
I20260928 17:50:07.122308 20403 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:50:07.122401 20403 leader_election.cc:302] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90fc26731db4c4bb93d09cd61f522ca; no voters:
I20260928 17:50:07.122555 20403 leader_election.cc:288] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 17:50:07.122597 20407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 3 FOLLOWER]: Leader election won for term 3
I20260928 17:50:07.122714 20407 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [term 3 LEADER]: Becoming Leader. State: Replica: e90fc26731db4c4bb93d09cd61f522ca, State: Running, Role: LEADER
I20260928 17:50:07.122812 20403 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:07.122793 20407 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } }
I20260928 17:50:07.123171 20408 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90fc26731db4c4bb93d09cd61f522ca. Latest consensus state: current_term: 3 leader_uuid: "e90fc26731db4c4bb93d09cd61f522ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } } }
I20260928 17:50:07.123291 20408 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:07.123183 20409 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e90fc26731db4c4bb93d09cd61f522ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90fc26731db4c4bb93d09cd61f522ca" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 36989 } } }
I20260928 17:50:07.123442 20409 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:07.123509 20415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:07.124253 20415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:07.124353 20415 catalog_manager.cc:1259] Loaded cluster ID: cfc818a997094d9d9cd7297d97192294
I20260928 17:50:07.124387 20415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:07.125494 20415 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:07.125658 20415 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Loaded TSK: 0
I20260928 17:50:07.125705 20415 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e90fc26731db4c4bb93d09cd61f522ca: Loaded TSK: 1
F20260928 17:50:07.125926 20415 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fce4b9199ad google::LogMessage::Fail() at ??:0
@ 0x7fce4b91c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fce4b9194de google::LogMessage::Flush() at ??:0
@ 0x7fce4b91aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fce4e086104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fce4e083d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fce538771cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fce5387748e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fce53877594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fce5383dc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fce53847ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fce4d7e70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fce4d7dc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fce4e6406db start_thread at ??:0
@ 0x7fce4a41761f clone at ??:0
W20260928 17:50:07.198704 20405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:07.198884 20405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:07.198906 20405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:07.200075 20405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:07.200119 20405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:07.200142 20405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:50:07.201534 20405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:37905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=43367
--tserver_master_addrs=127.3.222.62:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:07.201738 20405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:07.201941 20405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:07.207657 20444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:07.207726 20445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:07.207785 20405 server_base.cc:1034] running on GCE node
W20260928 17:50:07.207947 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:07.208124 20405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:07.208348 20405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:07.209492 20405 hybrid_clock.cc:648] HybridClock initialized: now 1790617807209482 us; error 33 us; skew 500 ppm
Sep 28 17:50:07 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:07.211858 20405 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:50:07.212267 20405 webserver.cc:460] Webserver started at http://127.3.222.1:43367/ using document root <none> and password file <none>
I20260928 17:50:07.212517 20405 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:07.212565 20405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:07.213811 20405 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:07.214391 20454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:07.214545 20405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:07.214620 20405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "ca2f00c73482424f9bd055baae51e4e7"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "1ebd54727164029558fb55bdf78e7d49"
server_key_iv: "ca8711ec4f01e757dc4ba20dfb639e72"
server_key_version: "encryptionkey@0"
I20260928 17:50:07.214928 20405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:07.230021 20405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:07.232023 20405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:07.232131 20405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:07.232348 20405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:07.232805 20461 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:50:07.234254 20405 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:50:07.234298 20405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:50:07.234321 20405 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:50:07.234905 20405 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:50:07.234946 20405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 17:50:07.235009 20461 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Bootstrap starting.
I20260928 17:50:07.242262 20405 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:37905
I20260928 17:50:07.242427 20568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:37905 every 8 connection(s)
I20260928 17:50:07.242753 20405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:50:07.245714 20461 log.cc:824] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:07.247648 20461 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:50:07.247881 20461 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Bootstrap complete.
I20260928 17:50:07.248592 20461 ts_tablet_manager.cc:1397] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260928 17:50:07.249516 20461 raft_consensus.cc:348] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:50:07.249728 20461 raft_consensus.cc:729] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2f00c73482424f9bd055baae51e4e7, State: Initialized, Role: FOLLOWER
I20260928 17:50:07.249848 20461 consensus_queue.cc:260] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:50:07.250051 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 20405
I20260928 17:50:07.250070 20461 ts_tablet_manager.cc:1428] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:07.250339 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:45401
--local_ip_for_outbound_sockets=127.3.222.2
--tserver_master_addrs=127.3.222.62:36989
--webserver_port=42493
--webserver_interface=127.3.222.2
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:07.301466 20571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:07.243441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:34981)
0928 17:50:07.243637 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.243698 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:07.244337 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:07.301264 (+ 56927us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
W20260928 17:50:07.301728 20569 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:36989 (0 consecutive failures): Network error: Failed to ping master at 127.3.222.62:36989: Client connection negotiation failed: client connection to 127.3.222.62:36989: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:50:07.302038 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.301936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:51585)
0928 17:50:07.301981 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.302017 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 17:50:07.302325 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.302254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:49907)
0928 17:50:07.302290 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.302310 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 17:50:07.302435 20569 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:50:07.328807 20574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:07.328999 20574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:07.329021 20574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:07.330132 20574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:07.330169 20574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:07.330193 20574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:50:07.331606 20574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:45401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=42493
--tserver_master_addrs=127.3.222.62:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:07.331827 20574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:07.332029 20574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:07.336726 20580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:07.336735 20579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:07.336740 20582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:07.336834 20574 server_base.cc:1034] running on GCE node
I20260928 17:50:07.337096 20574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:07.337283 20574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:07.338411 20574 hybrid_clock.cc:648] HybridClock initialized: now 1790617807338387 us; error 44 us; skew 500 ppm
Sep 28 17:50:07 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:07.340858 20574 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:50:07.341267 20574 webserver.cc:460] Webserver started at http://127.3.222.2:42493/ using document root <none> and password file <none>
I20260928 17:50:07.341475 20574 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:07.341518 20574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:07.342726 20574 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:07.343303 20589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:07.343444 20574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:07.343518 20574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "d1f523e4cf034c049da74bc97f85cd49"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "f2531a55e2cf1648ea436f509588ccc0"
server_key_iv: "1da764b56db9ec65b8cc935c93c9767a"
server_key_version: "encryptionkey@0"
I20260928 17:50:07.343854 20574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:07.362936 20574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:07.364981 20574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:07.365111 20574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:07.365311 20574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:07.365700 20596 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:50:07.367060 20574 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:50:07.367100 20574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:50:07.367130 20574 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:50:07.367739 20574 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:50:07.367770 20574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 17:50:07.367826 20596 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Bootstrap starting.
I20260928 17:50:07.374305 20574 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:45401
I20260928 17:50:07.374480 20703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:45401 every 8 connection(s)
I20260928 17:50:07.374702 20574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:50:07.375303 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 20574
I20260928 17:50:07.375619 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:44669
--local_ip_for_outbound_sockets=127.3.222.3
--tserver_master_addrs=127.3.222.62:36989
--webserver_port=44815
--webserver_interface=127.3.222.3
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:50:07.377275 20596 log.cc:824] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:07.377305 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.375287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:45481)
0928 17:50:07.377044 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.377136 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1713,"thread_start_us":1672,"threads_started":1}
W20260928 17:50:07.377529 20704 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:36989 (0 consecutive failures): Network error: Failed to ping master at 127.3.222.62:36989: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
I20260928 17:50:07.377780 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.377684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:46451)
0928 17:50:07.377730 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.377757 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:50:07.378078 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.377992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:34469)
0928 17:50:07.378031 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.378054 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 17:50:07.378158 20704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 17:50:07.379892 20596 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:50:07.380167 20596 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Bootstrap complete.
I20260928 17:50:07.381294 20596 ts_tablet_manager.cc:1397] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260928 17:50:07.382997 20596 raft_consensus.cc:348] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:50:07.383374 20596 raft_consensus.cc:729] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f523e4cf034c049da74bc97f85cd49, State: Initialized, Role: FOLLOWER
I20260928 17:50:07.383518 20596 consensus_queue.cc:260] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:50:07.383764 20596 ts_tablet_manager.cc:1428] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260928 17:50:07.383982 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.383889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:38009)
0928 17:50:07.383928 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.383955 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 17:50:07.384285 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.384200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:56069)
0928 17:50:07.384233 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.384255 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 17:50:07.454983 20706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:07.455193 20706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:07.455242 20706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:07.456378 20706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:07.456420 20706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:07.456454 20706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:50:07.457911 20706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:44669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=44815
--tserver_master_addrs=127.3.222.62:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:07.458133 20706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:07.458339 20706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:50:07.463236 20706 server_base.cc:1034] running on GCE node
W20260928 17:50:07.463187 20715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:07.463418 20714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:07.463263 20717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:07.463636 20706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:07.463886 20706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:07.465029 20706 hybrid_clock.cc:648] HybridClock initialized: now 1790617807465019 us; error 30 us; skew 500 ppm
Sep 28 17:50:07 dist-test-slave-w3bm krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:07.467411 20706 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:50:07.467862 20706 webserver.cc:460] Webserver started at http://127.3.222.3:44815/ using document root <none> and password file <none>
I20260928 17:50:07.468039 20706 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:07.468077 20706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:07.469347 20706 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:07.469925 20724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:07.470090 20706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:07.470188 20706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "814aab6152794923b2acb0df635c3595"
format_stamp: "Formatted at 2026-09-28 17:49:25 on dist-test-slave-w3bm"
server_key: "eafc98435edaf9e0703a9683dad86c76"
server_key_iv: "3f310fd374638ad3d8501dd2d66cf4ad"
server_key_version: "encryptionkey@0"
I20260928 17:50:07.470541 20706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:07.486897 20706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:07.488987 20706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:07.489106 20706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:07.489317 20706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:07.489598 20706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:07.489629 20706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:07.489650 20706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:07.489673 20706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:07.495008 20706 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:44669
I20260928 17:50:07.495074 20837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:44669 every 8 connection(s)
I20260928 17:50:07.495404 20706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:50:07.496331 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.495983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:49671)
0928 17:50:07.496121 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.496188 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
W20260928 17:50:07.496726 20838 heartbeater.cc:644] Failed to heartbeat to 127.3.222.62:36989 (0 consecutive failures): Network error: Failed to ping master at 127.3.222.62:36989: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
I20260928 17:50:07.496973 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.496885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:35989)
0928 17:50:07.496926 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.496952 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:50:07.497192 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:07.497142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:55923)
0928 17:50:07.497162 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:07.497180 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 17:50:07.497314 20838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 17:50:07.500625 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 20706
I20260928 17:50:08.302979 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:08.302771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:37379)
0928 17:50:08.302885 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:08.302921 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 17:50:08.384830 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:08.384625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:40673)
0928 17:50:08.384756 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:08.384788 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260928 17:50:08.497763 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:08.497597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:59667)
0928 17:50:08.497701 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:08.497721 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":50,"threads_started":1}
I20260928 17:50:08.673676 20845 raft_consensus.cc:482] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:50:08.673848 20845 raft_consensus.cc:504] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:50:08.674149 20845 leader_election.cc:288] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669), ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905)
Sep 28 17:50:08 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.1@KRBTEST.COM
Sep 28 17:50:08 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.3@KRBTEST.COM
I20260928 17:50:08.683569 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:08.674290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:44669 (local address 127.3.222.2:39357)
0928 17:50:08.674339 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:08.674385 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:08.674546 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:08.675250 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:08.675251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:08.675414 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:08.675465 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:08.675467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:08.676470 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:08.676471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:08.677269 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:08.677271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:08.677821 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:08.677822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:08.677924 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:08.678430 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:08.678447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:08.680317 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:08.682052 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:08.682053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:08.682053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:08.682242 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:08.682631 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:08.682632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:08.682632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:08.682687 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:08.683046 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:08.683053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:08.683169 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 17:50:08.683286 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 17:50:08.683454 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":86}
I20260928 17:50:08.683583 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:08.674462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:56095 (local address 127.3.222.1:37905)
0928 17:50:08.674651 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:08.674656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:08.674665 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:50:08.674740 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:08.674741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:08.674760 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:08.674831 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:08.675497 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:08.676379 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:08.677345 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:08.677729 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:08.677822 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:08.680054 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:08.680058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:08.680058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:08.680070 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:08.681807 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:08.682265 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:08.682269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:08.682269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:08.682411 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:08.682710 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:08.682711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:08.682711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:08.682978 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:08.683049 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:08.683358 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 17:50:08.683491 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":52,"threads_started":1}
I20260928 17:50:08.683638 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:08.674482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:39357 (local address 127.3.222.3:44669)
0928 17:50:08.674627 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:08.674638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:08.674665 (+ 27us) server_negotiation.cc:407] Connection header received
0928 17:50:08.675088 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:08.675089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:08.675124 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:08.675254 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:08.675553 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:08.676394 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:08.677379 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:08.677753 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:08.677843 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:08.680402 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:08.680407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:08.680407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:08.680417 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:08.681963 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:08.682360 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:08.682363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:08.682363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:08.682555 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:08.682749 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:08.682750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:08.682750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:08.682980 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:08.683049 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:08.683409 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 17:50:08.683534 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":50,"threads_started":1}
I20260928 17:50:08.683624 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:08.674352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37905 (local address 127.3.222.2:56095)
0928 17:50:08.674458 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:08.674468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:08.674546 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:08.674888 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:08.674892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:08.675187 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:08.675403 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:08.675409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:08.676467 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:08.676468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:08.677270 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:08.677272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:08.677811 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:08.677812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:08.677921 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:08.678464 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:08.678468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:08.679957 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:08.681874 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:08.681881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:08.681883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:08.682164 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:08.682517 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:08.682518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:08.682518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:08.682650 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:08.683046 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:08.683053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:08.683169 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 17:50:08.683284 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 17:50:08.683520 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":137,"thread_start_us":47,"threads_started":1}
I20260928 17:50:08.684618 20523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e2cfa3d8a6438aae11c70657051495" candidate_uuid: "d1f523e4cf034c049da74bc97f85cd49" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca2f00c73482424f9bd055baae51e4e7" is_pre_election: true
I20260928 17:50:08.684744 20523 raft_consensus.cc:2454] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f523e4cf034c049da74bc97f85cd49 in term 1.
W20260928 17:50:08.684921 20591 leader_election.cc:341] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669): Not found: Tablet not found: d1e2cfa3d8a6438aae11c70657051495
I20260928 17:50:08.685020 20593 leader_election.cc:302] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca2f00c73482424f9bd055baae51e4e7, d1f523e4cf034c049da74bc97f85cd49; no voters: 814aab6152794923b2acb0df635c3595
I20260928 17:50:08.685122 20845 raft_consensus.cc:2781] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 17:50:08.685184 20845 raft_consensus.cc:482] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:50:08.685202 20845 raft_consensus.cc:3037] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:50:08.686265 20845 raft_consensus.cc:504] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:50:08.686386 20845 leader_election.cc:288] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 2 election: Requested vote from peers 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669), ca2f00c73482424f9bd055baae51e4e7 (127.3.222.1:37905)
I20260928 17:50:08.686517 20523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e2cfa3d8a6438aae11c70657051495" candidate_uuid: "d1f523e4cf034c049da74bc97f85cd49" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca2f00c73482424f9bd055baae51e4e7"
I20260928 17:50:08.686594 20523 raft_consensus.cc:3037] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 1 FOLLOWER]: Advancing to term 2
W20260928 17:50:08.686676 20591 leader_election.cc:341] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669): Not found: Tablet not found: d1e2cfa3d8a6438aae11c70657051495
I20260928 17:50:08.687682 20523 raft_consensus.cc:2454] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f523e4cf034c049da74bc97f85cd49 in term 2.
I20260928 17:50:08.687856 20593 leader_election.cc:302] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca2f00c73482424f9bd055baae51e4e7, d1f523e4cf034c049da74bc97f85cd49; no voters: 814aab6152794923b2acb0df635c3595
I20260928 17:50:08.687942 20845 raft_consensus.cc:2781] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:50:08.688069 20845 raft_consensus.cc:686] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [term 2 LEADER]: Becoming Leader. State: Replica: d1f523e4cf034c049da74bc97f85cd49, State: Running, Role: LEADER
I20260928 17:50:08.688151 20845 consensus_queue.cc:237] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:50:08.688385 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:08.688231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:49451)
0928 17:50:08.688334 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:08.688360 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 17:50:09.068176 20523 raft_consensus.cc:1261] T d1e2cfa3d8a6438aae11c70657051495 P ca2f00c73482424f9bd055baae51e4e7 [term 2 FOLLOWER]: Refusing update from remote peer d1f523e4cf034c049da74bc97f85cd49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 17:50:09.068547 20845 consensus_queue.cc:1035] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 17:50:09.069424 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:09.069223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:56299)
0928 17:50:09.069342 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:09.069385 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":48,"threads_started":1}
I20260928 17:50:09.070169 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:09.070075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:45093)
0928 17:50:09.070113 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:09.070137 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 17:50:09.070350 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:09.070212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:37279)
0928 17:50:09.070297 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:09.070321 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 17:50:09.070363 20591 consensus_peers.cc:487] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49 -> Peer 814aab6152794923b2acb0df635c3595 (127.3.222.3:44669): Couldn't send request to peer 814aab6152794923b2acb0df635c3595. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1e2cfa3d8a6438aae11c70657051495. This is attempt 1: this message will repeat every 5th retry.
I20260928 17:50:09.498378 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:09.498167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:48727)
0928 17:50:09.498303 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:09.498339 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260928 17:50:09.532233 20858 ts_tablet_manager.cc:927] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Initiating tablet copy from peer d1f523e4cf034c049da74bc97f85cd49 (127.3.222.2:45401)
I20260928 17:50:09.532521 20858 tablet_copy_client.cc:316] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: tablet copy: Beginning tablet copy session from remote peer at address 127.3.222.2:45401
Sep 28 17:50:09 dist-test-slave-w3bm krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.2@KRBTEST.COM
I20260928 17:50:09.539656 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:09.532864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:50039 (local address 127.3.222.2:45401)
0928 17:50:09.532969 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:09.532973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:09.532982 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:50:09.533006 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:09.533007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:09.533020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:09.533081 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:09.533519 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:09.534187 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:09.534919 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:09.535224 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:09.535283 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:09.537205 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:09.537215 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:09.537215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:09.537222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:09.538624 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:09.538951 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:09.538953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:09.538953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:09.539002 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:09.539134 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:09.539134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:09.539135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:09.539293 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:09.539355 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:09.539522 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 17:50:09.539590 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":41,"threads_started":1}
I20260928 17:50:09.539657 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:09.532770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:45401 (local address 127.3.222.3:50039)
0928 17:50:09.532835 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:09.532853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:09.532913 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:09.533091 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:09.533093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:09.533384 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:09.533455 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:09.533458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:09.534261 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:09.534262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:09.534858 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:09.534860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:09.535290 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:09.535292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:09.535375 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:09.535797 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:09.535801 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:09.537126 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:09.538690 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:09.538697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:09.538698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:09.538887 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:09.539046 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:09.539047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:09.539047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:09.539096 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:09.539368 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:09.539374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:09.539421 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 17:50:09.539492 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 17:50:09.539580 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 17:50:09.540066 20678 tablet_copy_service.cc:140] P d1f523e4cf034c049da74bc97f85cd49: Received BeginTabletCopySession request for tablet d1e2cfa3d8a6438aae11c70657051495 from peer 814aab6152794923b2acb0df635c3595 ({username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:50039)
I20260928 17:50:09.540128 20678 tablet_copy_service.cc:161] P d1f523e4cf034c049da74bc97f85cd49: Beginning new tablet copy session on tablet d1e2cfa3d8a6438aae11c70657051495 from peer 814aab6152794923b2acb0df635c3595 at {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:50039: session id = 814aab6152794923b2acb0df635c3595-d1e2cfa3d8a6438aae11c70657051495
I20260928 17:50:09.540572 20678 tablet_copy_source_session.cc:205] T d1e2cfa3d8a6438aae11c70657051495 P d1f523e4cf034c049da74bc97f85cd49: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 17:50:09.541448 20858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e2cfa3d8a6438aae11c70657051495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:09.543880 20858 tablet_copy_client.cc:791] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: tablet copy: Starting download of 0 data blocks...
I20260928 17:50:09.544005 20858 tablet_copy_client.cc:655] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: tablet copy: Starting download of 1 WAL segments...
I20260928 17:50:09.544801 20858 tablet_copy_client.cc:531] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 17:50:09.545460 20858 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Bootstrap starting.
I20260928 17:50:09.546761 20858 log.cc:824] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:09.549015 20858 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:50:09.549196 20858 tablet_bootstrap.cc:492] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Bootstrap complete.
I20260928 17:50:09.549304 20858 ts_tablet_manager.cc:1397] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 17:50:09.550176 20858 raft_consensus.cc:348] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:50:09.550311 20858 raft_consensus.cc:729] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814aab6152794923b2acb0df635c3595, State: Initialized, Role: FOLLOWER
I20260928 17:50:09.550427 20858 consensus_queue.cc:260] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f523e4cf034c049da74bc97f85cd49" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 45401 } } peers { permanent_uuid: "814aab6152794923b2acb0df635c3595" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 44669 } } peers { permanent_uuid: "ca2f00c73482424f9bd055baae51e4e7" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37905 } }
I20260928 17:50:09.550681 20858 ts_tablet_manager.cc:1428] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20260928 17:50:09.550921 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:09.550839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:38325)
0928 17:50:09.550869 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:09.550894 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 17:50:09.550961 20678 tablet_copy_service.cc:342] P d1f523e4cf034c049da74bc97f85cd49: Request end of tablet copy session 814aab6152794923b2acb0df635c3595-d1e2cfa3d8a6438aae11c70657051495 received from {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:50039
I20260928 17:50:09.551004 20678 tablet_copy_service.cc:433] P d1f523e4cf034c049da74bc97f85cd49: ending tablet copy session 814aab6152794923b2acb0df635c3595-d1e2cfa3d8a6438aae11c70657051495 on tablet d1e2cfa3d8a6438aae11c70657051495 with peer 814aab6152794923b2acb0df635c3595
I20260928 17:50:09.551254 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:09.551152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:58125)
0928 17:50:09.551189 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:09.551231 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 17:50:10.070721 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:10.070508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:57859)
0928 17:50:10.070619 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:10.070665 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260928 17:50:10.070931 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:10.070728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:44747)
0928 17:50:10.070839 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:10.070884 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260928 17:50:10.128714 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:10.128597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:50331)
0928 17:50:10.128646 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:10.128675 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 17:50:10.128737 20792 raft_consensus.cc:1203] T d1e2cfa3d8a6438aae11c70657051495 P 814aab6152794923b2acb0df635c3595 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 17:50:10.129279 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:10.129108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:56403)
0928 17:50:10.129191 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:10.129226 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":48,"threads_started":1}
I20260928 17:50:11.071570 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:11.071357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:55963)
0928 17:50:11.071486 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:11.071522 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260928 17:50:11.129359 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:11.129134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:42329)
0928 17:50:11.129272 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:11.129310 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260928 17:50:11.129846 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:11.129678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:39847)
0928 17:50:11.129779 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:11.129811 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
I20260928 17:50:12.072232 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:12.072035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:37489)
0928 17:50:12.072152 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:12.072184 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260928 17:50:12.130061 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:12.129822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:50829)
0928 17:50:12.129970 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:12.130013 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260928 17:50:12.130476 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:12.130205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:34649)
0928 17:50:12.130335 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:12.130429 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260928 17:50:13.073038 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:13.072775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:57709)
0928 17:50:13.072943 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:13.072992 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
I20260928 17:50:13.130928 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:13.130629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:59629)
0928 17:50:13.130809 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:13.130856 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260928 17:50:13.131261 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:13.131009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:34051)
0928 17:50:13.131137 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:13.131184 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260928 17:50:13.131428 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:13.131277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:55453)
0928 17:50:13.131319 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:13.131351 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:50:14.073669 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:14.073433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:39989)
0928 17:50:14.073572 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:14.073611 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260928 17:50:14.131877 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:14.131650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:49529)
0928 17:50:14.131764 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:14.131817 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260928 17:50:14.132045 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:14.131812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:47935)
0928 17:50:14.131966 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:14.132003 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260928 17:50:15.074252 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:15.074054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:47803)
0928 17:50:15.074171 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:15.074206 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 17:50:15.132575 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:15.132370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:35235)
0928 17:50:15.132492 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:15.132524 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260928 17:50:15.132716 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:15.132489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:35629)
0928 17:50:15.132639 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:15.132667 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260928 17:50:16.074903 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:16.074681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:34743)
0928 17:50:16.074819 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:16.074856 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260928 17:50:16.133364 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:16.133151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:36837)
0928 17:50:16.133269 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:16.133307 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 17:50:16.133364 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:16.133133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:60905)
0928 17:50:16.133296 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:16.133327 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260928 17:50:17.075623 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:17.075372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:36247)
0928 17:50:17.075534 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:17.075575 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260928 17:50:17.134023 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:17.133800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:60207)
0928 17:50:17.133925 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:17.133961 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260928 17:50:17.134030 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:17.133815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:54297)
0928 17:50:17.133956 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:17.133989 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260928 17:50:18.076267 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:18.076064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:42301)
0928 17:50:18.076181 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:18.076219 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260928 17:50:18.134773 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:18.134551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:38447)
0928 17:50:18.134687 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:18.134724 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260928 17:50:18.134848 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:18.134585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:42815)
0928 17:50:18.134720 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:18.134760 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260928 17:50:19.076920 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:19.076694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:34911)
0928 17:50:19.076833 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:19.076871 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260928 17:50:19.135542 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:19.135320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:60819)
0928 17:50:19.135457 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:19.135494 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260928 17:50:19.135692 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:19.135321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:50001)
0928 17:50:19.135458 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:19.135494 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260928 17:50:20.077476 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:20.077296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:48493)
0928 17:50:20.077401 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:20.077430 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260928 17:50:20.136260 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:20.136013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:36867)
0928 17:50:20.136150 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:20.136193 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260928 17:50:20.136287 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:20.136073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:56929)
0928 17:50:20.136192 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:20.136227 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260928 17:50:21.078068 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:21.077865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:49155)
0928 17:50:21.077988 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:21.078020 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260928 17:50:21.136952 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:21.136715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:54819)
0928 17:50:21.136870 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:21.136905 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260928 17:50:21.136951 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:21.136760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:35989)
0928 17:50:21.136875 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:21.136909 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260928 17:50:22.078740 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:22.078486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.1:53677)
0928 17:50:22.078632 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:22.078681 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260928 17:50:22.137650 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:22.137437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.2:41787)
0928 17:50:22.137582 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:22.137612 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260928 17:50:22.137650 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:22.137438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:36989 (local address 127.3.222.3:43125)
0928 17:50:22.137556 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:22.137593 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.62:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 17:50:22.506070 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 20405
I20260928 17:50:22.510056 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 20574
I20260928 17:50:22.515802 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 20706
2026-09-28T17:50:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58217 ms)
[----------] 30 tests from SecurityITest (222868 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:22 dist-test-slave-w3bm krb5kdc[20907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:22 dist-test-slave-w3bm krb5kdc[20907](info): set up 2 sockets
Sep 28 17:50:22 dist-test-slave-w3bm krb5kdc[20907](info): commencing operation
krb5kdc: starting...
W20260928 17:50:23.470639 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:23 dist-test-slave-w3bm krb5kdc[20907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:23.587889 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41813
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:38529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:23.666169 20920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:23.666368 20920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:23.666397 20920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:23.667560 20920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:23.667600 20920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:23.667618 20920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:23.667634 20920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:23.667649 20920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:23.669100 20920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:38529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:23.669345 20920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:23.669541 20920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:50:23.674420 20920 server_base.cc:1034] running on GCE node
W20260928 17:50:23.674427 20925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:23.674566 20926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:23.674576 20928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:23.674913 20920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:23.675154 20920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:23.676311 20920 hybrid_clock.cc:648] HybridClock initialized: now 1790617823676290 us; error 34 us; skew 500 ppm
Sep 28 17:50:23 dist-test-slave-w3bm krb5kdc[20907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:23.678632 20920 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:23.679070 20920 webserver.cc:460] Webserver started at http://127.3.222.62:41169/ using document root <none> and password file <none>
I20260928 17:50:23.679315 20920 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:23.679373 20920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:23.679504 20920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:23.680533 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "b451ec2da0de489c93888a83f684609c"
format_stamp: "Formatted at 2026-09-28 17:50:23 on dist-test-slave-w3bm"
server_key: "28d65e938329c88c9e1138eb16cc0499"
server_key_iv: "240bc18b4f82acd546d4f46b13408e00"
server_key_version: "encryptionkey@0"
I20260928 17:50:23.680861 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "b451ec2da0de489c93888a83f684609c"
format_stamp: "Formatted at 2026-09-28 17:50:23 on dist-test-slave-w3bm"
server_key: "28d65e938329c88c9e1138eb16cc0499"
server_key_iv: "240bc18b4f82acd546d4f46b13408e00"
server_key_version: "encryptionkey@0"
I20260928 17:50:23.682144 20920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:50:23.682825 20935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:23.682979 20920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:50:23.683061 20920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "b451ec2da0de489c93888a83f684609c"
format_stamp: "Formatted at 2026-09-28 17:50:23 on dist-test-slave-w3bm"
server_key: "28d65e938329c88c9e1138eb16cc0499"
server_key_iv: "240bc18b4f82acd546d4f46b13408e00"
server_key_version: "encryptionkey@0"
I20260928 17:50:23.683115 20920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:23.698902 20920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:23.700716 20920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:23.700820 20920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:23.704355 20920 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:41813
I20260928 17:50:23.704401 20987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:41813 every 8 connection(s)
I20260928 17:50:23.704689 20920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:23.705502 20988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:23.707716 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c: Bootstrap starting.
I20260928 17:50:23.708246 20988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:23.708490 20988 log.cc:824] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:23.709103 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c: No bootstrap required, opened a new log
I20260928 17:50:23.710701 20988 raft_consensus.cc:348] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451ec2da0de489c93888a83f684609c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41813 } }
I20260928 17:50:23.710834 20988 raft_consensus.cc:374] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:23.710853 20988 raft_consensus.cc:729] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b451ec2da0de489c93888a83f684609c, State: Initialized, Role: FOLLOWER
I20260928 17:50:23.710989 20988 consensus_queue.cc:260] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451ec2da0de489c93888a83f684609c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41813 } }
I20260928 17:50:23.711054 20988 raft_consensus.cc:388] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:23.711074 20988 raft_consensus.cc:482] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:23.711104 20988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:23.711619 20988 raft_consensus.cc:504] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451ec2da0de489c93888a83f684609c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41813 } }
I20260928 17:50:23.711707 20988 leader_election.cc:302] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b451ec2da0de489c93888a83f684609c; no voters:
I20260928 17:50:23.711854 20988 leader_election.cc:288] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:23.711879 20991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:23.712000 20991 raft_consensus.cc:686] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [term 1 LEADER]: Becoming Leader. State: Replica: b451ec2da0de489c93888a83f684609c, State: Running, Role: LEADER
I20260928 17:50:23.712093 20988 sys_catalog.cc:564] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:23.712124 20991 consensus_queue.cc:237] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451ec2da0de489c93888a83f684609c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41813 } }
I20260928 17:50:23.712455 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 20920
I20260928 17:50:23.712559 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:23.712775 3960 external_mini_cluster.cc:1428] Setting key 02fc74b9a903e2a6b43b12c13ce62eb3
I20260928 17:50:23.713297 20993 sys_catalog.cc:455] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b451ec2da0de489c93888a83f684609c. Latest consensus state: current_term: 1 leader_uuid: "b451ec2da0de489c93888a83f684609c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451ec2da0de489c93888a83f684609c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41813 } } }
I20260928 17:50:23.713304 20992 sys_catalog.cc:455] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b451ec2da0de489c93888a83f684609c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451ec2da0de489c93888a83f684609c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41813 } } }
I20260928 17:50:23.713447 20993 sys_catalog.cc:458] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:23.713454 20992 sys_catalog.cc:458] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:23.713860 21009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:23.714318 21009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:23.716240 21009 catalog_manager.cc:1347] Generated new cluster ID: 83846d670dca40f081fbdf750610d71d
I20260928 17:50:23.716315 21009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:50:23 dist-test-slave-w3bm krb5kdc[20907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:23.723459 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:23.713382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.3.222.62:41813)
0928 17:50:23.713571 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:23.713581 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:23.713613 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:50:23.713748 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:23.713756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:23.713814 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:23.713948 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:23.714622 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:23.715786 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:23.716776 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:23.717427 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:23.717541 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:23.720144 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:23.720148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:23.720149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:23.720160 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:23.721934 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:23.722301 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:23.722304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:23.722304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:23.722407 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:23.722589 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:23.722589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:23.722590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:23.722863 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:23.723044 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:23.723106 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 17:50:23.723257 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260928 17:50:23.726967 21009 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:23.727515 21009 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:23.733801 21009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c: Generated new TSK 0
I20260928 17:50:23.733982 21009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:50:23.760385 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41813
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:38529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:23.838982 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:23.839183 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:23.839232 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:23.840358 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:23.840396 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:23.840430 21014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:50:23.841851 21014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:38529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:23.842079 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:23.842280 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:23.847229 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:23.847373 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:23.847404 21014 server_base.cc:1034] running on GCE node
W20260928 17:50:23.847290 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:23.847702 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:23.847983 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:23.849143 21014 hybrid_clock.cc:648] HybridClock initialized: now 1790617823849123 us; error 35 us; skew 500 ppm
Sep 28 17:50:23 dist-test-slave-w3bm krb5kdc[20907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:23.851502 21014 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:50:23.851879 21014 webserver.cc:460] Webserver started at http://127.3.222.1:38183/ using document root <none> and password file <none>
I20260928 17:50:23.852051 21014 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:23.852088 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:23.852173 21014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:23.853133 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "983bd752223e49e092368be390489d4f"
format_stamp: "Formatted at 2026-09-28 17:50:23 on dist-test-slave-w3bm"
server_key: "32ae831449abfc87cd395bd3bbab2b6f"
server_key_iv: "0b32ae4a09eaa6eb31ca7763ac3514f9"
server_key_version: "encryptionkey@0"
I20260928 17:50:23.853413 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "983bd752223e49e092368be390489d4f"
format_stamp: "Formatted at 2026-09-28 17:50:23 on dist-test-slave-w3bm"
server_key: "32ae831449abfc87cd395bd3bbab2b6f"
server_key_iv: "0b32ae4a09eaa6eb31ca7763ac3514f9"
server_key_version: "encryptionkey@0"
I20260928 17:50:23.854571 21014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:23.855290 21029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:23.855449 21014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:23.855521 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "983bd752223e49e092368be390489d4f"
format_stamp: "Formatted at 2026-09-28 17:50:23 on dist-test-slave-w3bm"
server_key: "32ae831449abfc87cd395bd3bbab2b6f"
server_key_iv: "0b32ae4a09eaa6eb31ca7763ac3514f9"
server_key_version: "encryptionkey@0"
I20260928 17:50:23.855587 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:23.885495 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:23.887740 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:23.887887 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:23.888103 21014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:23.888402 21014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:23.888435 21014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:23.888463 21014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:23.888486 21014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:23.893816 21014 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:41389
I20260928 17:50:23.893867 21142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:41389 every 8 connection(s)
I20260928 17:50:23.894164 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:50:23.895390 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 21014
I20260928 17:50:23.895476 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:50:23.895648 3960 external_mini_cluster.cc:1428] Setting key 1884a93e6381d6ade71371f991810145
Sep 28 17:50:23 dist-test-slave-w3bm krb5kdc[20907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
I20260928 17:50:23.903489 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:23.894841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:34957 (local address 127.3.222.62:41813)
0928 17:50:23.894901 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:23.894903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:23.895106 (+ 203us) server_negotiation.cc:407] Connection header received
0928 17:50:23.895709 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:23.895711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:23.895727 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:23.895781 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:23.896807 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:23.897303 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:23.898410 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:23.898686 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:23.898754 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:23.900922 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:23.900923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:23.900924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:23.900930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:23.902125 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:23.902574 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:23.902575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:23.902575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:23.902621 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:23.902871 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:23.902871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:23.902871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:23.902967 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:23.903011 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:23.903353 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 17:50:23.903424 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 17:50:23.903728 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:23.894829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41813 (local address 127.3.222.1:34957)
0928 17:50:23.894994 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:23.895030 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:23.895612 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:23.895869 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:23.895874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:23.896510 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:23.896712 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:23.896717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:23.897384 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:23.897385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:23.898314 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:23.898317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:23.898763 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:23.898764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:23.898896 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:23.899375 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:23.899383 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:23.900832 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:23.902209 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:23.902216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:23.902218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:23.902505 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:23.902680 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:23.902680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:23.902681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:23.902813 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:23.903018 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:23.903023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:23.903191 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 17:50:23.903338 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 17:50:23.903500 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260928 17:50:23.904207 21143 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41813
I20260928 17:50:23.904302 21143 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:23.904444 21143 heartbeater.cc:511] Master 127.3.222.62:41813 requested a full tablet report, sending...
I20260928 17:50:23.904955 20952 ts_manager.cc:194] Registered new tserver with Master: 983bd752223e49e092368be390489d4f (127.3.222.1:41389)
I20260928 17:50:23.905553 20952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:34957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:50:23.922231 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41813
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:38529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:24.001159 21148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:24.001353 21148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:24.001384 21148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:24.002498 21148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:24.002537 21148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:24.002570 21148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:50:24.004025 21148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:38529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:24.004252 21148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:24.004449 21148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:24.009174 21154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.009300 21148 server_base.cc:1034] running on GCE node
W20260928 17:50:24.009311 21156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:24.009356 21153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.009598 21148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:24.009843 21148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:24.010984 21148 hybrid_clock.cc:648] HybridClock initialized: now 1790617824010969 us; error 27 us; skew 500 ppm
Sep 28 17:50:24 dist-test-slave-w3bm krb5kdc[20907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:24.013547 21148 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:50:24.014014 21148 webserver.cc:460] Webserver started at http://127.3.222.2:34719/ using document root <none> and password file <none>
I20260928 17:50:24.014216 21148 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:24.014261 21148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:24.014384 21148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:24.015452 21148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "6475b9ac70de429285c9859b1daa0f4c"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "acea0596c8833b4e83997fdb74472a57"
server_key_iv: "90cfbfe0e777a5fab066327ca6f73933"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.015772 21148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "6475b9ac70de429285c9859b1daa0f4c"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "acea0596c8833b4e83997fdb74472a57"
server_key_iv: "90cfbfe0e777a5fab066327ca6f73933"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.016996 21148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:24.017773 21163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.017958 21148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:24.018059 21148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "6475b9ac70de429285c9859b1daa0f4c"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "acea0596c8833b4e83997fdb74472a57"
server_key_iv: "90cfbfe0e777a5fab066327ca6f73933"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.018129 21148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:24.024852 21148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:24.026520 21148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:24.026631 21148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:24.026818 21148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:24.027081 21148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:24.027112 21148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.027137 21148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:24.027160 21148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.032389 21148 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:33117
I20260928 17:50:24.032433 21276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:33117 every 8 connection(s)
I20260928 17:50:24.032713 21148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:50:24.036442 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 21148
I20260928 17:50:24.036523 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:50:24.036692 3960 external_mini_cluster.cc:1428] Setting key 86c02fbce2a91164a9b355f15e6d007d
Sep 28 17:50:24 dist-test-slave-w3bm krb5kdc[20907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:24.041702 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.033385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:53267 (local address 127.3.222.62:41813)
0928 17:50:24.033439 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.033441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.033643 (+ 202us) server_negotiation.cc:407] Connection header received
0928 17:50:24.034169 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.034171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.034184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.034234 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:24.035108 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.035654 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.036640 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.036876 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.036935 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.039066 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:24.039067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:24.039068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:24.039074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:24.040349 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:24.040821 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:24.040822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:24.040822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:24.040876 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:24.041121 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:24.041121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:24.041121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:24.041221 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:24.041266 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.041564 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.041637 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:50:24.041945 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.033364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41813 (local address 127.3.222.2:53267)
0928 17:50:24.033504 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.033537 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.034064 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.034300 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.034305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.034835 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:24.035015 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.035020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.035730 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.035731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.036528 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.036531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.036941 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.036942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.037081 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.037583 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:24.037591 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:24.038973 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:24.040427 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:24.040434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:24.040437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:24.040719 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:24.040936 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:24.040937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:24.040937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:24.041066 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:24.041275 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:24.041280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:24.041440 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.041554 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.041712 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260928 17:50:24.042536 21277 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41813
I20260928 17:50:24.042639 21277 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:24.042833 21277 heartbeater.cc:511] Master 127.3.222.62:41813 requested a full tablet report, sending...
I20260928 17:50:24.043291 20952 ts_manager.cc:194] Registered new tserver with Master: 6475b9ac70de429285c9859b1daa0f4c (127.3.222.2:33117)
I20260928 17:50:24.043733 20952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:53267
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:50:24.062960 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41813
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:38529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:24.141348 21282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:24.141538 21282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:24.141561 21282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:24.142675 21282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:24.142717 21282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:24.142742 21282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:50:24.144194 21282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:38529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:24.144402 21282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:24.144627 21282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:24.149389 21288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:24.149566 21290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:24.149523 21287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.149559 21282 server_base.cc:1034] running on GCE node
I20260928 17:50:24.149819 21282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:24.150048 21282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:24.151201 21282 hybrid_clock.cc:648] HybridClock initialized: now 1790617824151184 us; error 32 us; skew 500 ppm
Sep 28 17:50:24 dist-test-slave-w3bm krb5kdc[20907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:24.153703 21282 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:50:24.154150 21282 webserver.cc:460] Webserver started at http://127.3.222.3:39731/ using document root <none> and password file <none>
I20260928 17:50:24.154369 21282 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:24.154417 21282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:24.154546 21282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:24.155658 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "eee750e1da2a4503a54490274082db9f"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "56f801aeb7ae988fce53ea06a40fe99b"
server_key_iv: "b8344209f5f112810118f658c588f2ce"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.155979 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "eee750e1da2a4503a54490274082db9f"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "56f801aeb7ae988fce53ea06a40fe99b"
server_key_iv: "b8344209f5f112810118f658c588f2ce"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.157217 21282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:24.157974 21297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.158130 21282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:24.158222 21282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "eee750e1da2a4503a54490274082db9f"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "56f801aeb7ae988fce53ea06a40fe99b"
server_key_iv: "b8344209f5f112810118f658c588f2ce"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.158339 21282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:24.168054 21282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:24.169770 21282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:24.169894 21282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:24.170096 21282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:24.170389 21282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:24.170423 21282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.170449 21282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:24.170465 21282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.175850 21282 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:35409
I20260928 17:50:24.175913 21410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:35409 every 8 connection(s)
I20260928 17:50:24.176191 21282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:50:24.177399 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 21282
I20260928 17:50:24.177498 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:50:24.177690 3960 external_mini_cluster.cc:1428] Setting key 7cd22b849d84b2a5e479c02c8e25c3b1
Sep 28 17:50:24 dist-test-slave-w3bm krb5kdc[20907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:24.184971 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.176806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:46113 (local address 127.3.222.62:41813)
0928 17:50:24.176856 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.176858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.177017 (+ 159us) server_negotiation.cc:407] Connection header received
0928 17:50:24.177610 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.177611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.177623 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.177709 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:24.178854 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.179419 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.180396 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.180584 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.180638 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.182498 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:24.182499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:24.182500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:24.182506 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:24.183778 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:24.184122 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:24.184124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:24.184124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:24.184175 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:24.184387 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:24.184388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:24.184388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:24.184494 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:24.184549 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.184804 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.184881 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:50:24.185156 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.176771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41813 (local address 127.3.222.3:46113)
0928 17:50:24.176903 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.176943 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.177512 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.177813 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.177818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.178533 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:24.178764 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.178770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.179498 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.179499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.180310 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.180312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.180645 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.180645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.180752 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.181174 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:24.181182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:24.182421 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:24.183848 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:24.183855 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:24.183857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:24.184059 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:24.184230 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:24.184230 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:24.184231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:24.184335 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:24.184547 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:24.184553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:24.184684 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.184778 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.184953 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260928 17:50:24.185693 21411 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41813
I20260928 17:50:24.185772 21411 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:24.185909 21411 heartbeater.cc:511] Master 127.3.222.62:41813 requested a full tablet report, sending...
I20260928 17:50:24.186234 20952 ts_manager.cc:194] Registered new tserver with Master: eee750e1da2a4503a54490274082db9f (127.3.222.3:35409)
I20260928 17:50:24.186602 20952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:46113
I20260928 17:50:24.189231 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:50:24.195801 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.190455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.3.222.62:41813)
0928 17:50:24.190495 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.190497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.190578 (+ 81us) server_negotiation.cc:407] Connection header received
0928 17:50:24.190626 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.190627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.190634 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.190676 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:24.191201 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.191714 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.192418 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.192604 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.192657 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.193725 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:24.193726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:24.193727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:24.193732 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:24.194836 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:24.195111 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:24.195112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:24.195112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:24.195151 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:24.195346 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:24.195346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:24.195347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:24.195478 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:24.195552 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.195682 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.195735 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:50:24.197569 20952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:50:24.199126 20952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:50:24.205101 21421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.201204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.3.222.3:35409)
0928 17:50:24.201810 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.201814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.201822 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:50:24.201857 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.201859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.201873 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.201937 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:24.202237 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.203291 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.204251 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.204731 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.204792 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.204852 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.204976 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.205019 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":83,"threads_started":1}
I20260928 17:50:24.205093 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.201126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35409 (local address 127.0.0.1:45522)
0928 17:50:24.201459 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.201470 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.201623 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.202050 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.202051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.202054 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:24.202153 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.202155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.203379 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.203380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.204184 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.204186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.204802 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.204803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.204880 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.204893 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.204955 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.205008 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260928 17:50:24.205201 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.201126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:33117 (local address 127.0.0.1:58042)
0928 17:50:24.201313 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.201351 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.201410 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.201693 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.201696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.201698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:24.201752 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.201753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.202820 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.202821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.204030 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.204033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.204984 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.204985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.205047 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.205051 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.205091 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.205131 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260928 17:50:24.205375 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.201344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:41389 (local address 127.0.0.1:47522)
0928 17:50:24.201509 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.201517 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.201553 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.201855 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.201856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.201857 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:24.201894 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.201895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.203114 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.203115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.204207 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.204209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.205034 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.205034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.205084 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.205087 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.205113 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.205321 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260928 17:50:24.205494 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.201291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.3.222.2:33117)
0928 17:50:24.201441 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.201446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.201457 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:50:24.201498 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.201501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.201516 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.201592 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:24.201820 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.202727 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.204120 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.204828 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.204893 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.204950 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.205359 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.205404 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260928 17:50:24.205785 21345 tablet_service.cc:1467] Processing CreateTablet for tablet a9638a611c85481e84202d96b40ec3ce (DEFAULT_TABLE table=test-table [id=5430c51ed0ad45da86c78d116af1c2f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:24.206022 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.201560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.3.222.1:41389)
0928 17:50:24.201716 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.201721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.201730 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:50:24.201769 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.201771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.201784 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.201860 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:24.201939 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.203011 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.204290 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.204954 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.205824 (+ 870us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.205889 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.205910 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.205944 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260928 17:50:24.206168 21345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9638a611c85481e84202d96b40ec3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:24.206182 21211 tablet_service.cc:1467] Processing CreateTablet for tablet a9638a611c85481e84202d96b40ec3ce (DEFAULT_TABLE table=test-table [id=5430c51ed0ad45da86c78d116af1c2f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:24.206497 21211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9638a611c85481e84202d96b40ec3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:24.206581 21077 tablet_service.cc:1467] Processing CreateTablet for tablet a9638a611c85481e84202d96b40ec3ce (DEFAULT_TABLE table=test-table [id=5430c51ed0ad45da86c78d116af1c2f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:24.206872 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9638a611c85481e84202d96b40ec3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:24.209252 21425 tablet_bootstrap.cc:492] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f: Bootstrap starting.
I20260928 17:50:24.209635 21427 tablet_bootstrap.cc:492] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: Bootstrap starting.
I20260928 17:50:24.209969 21425 tablet_bootstrap.cc:654] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:24.210189 21427 tablet_bootstrap.cc:654] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:24.210453 21425 log.cc:824] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:24.210614 21427 log.cc:824] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:24.210934 21426 tablet_bootstrap.cc:492] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: Bootstrap starting.
I20260928 17:50:24.211191 21425 tablet_bootstrap.cc:492] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f: No bootstrap required, opened a new log
I20260928 17:50:24.211287 21425 ts_tablet_manager.cc:1397] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:24.211303 21427 tablet_bootstrap.cc:492] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: No bootstrap required, opened a new log
I20260928 17:50:24.211359 21427 ts_tablet_manager.cc:1397] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:24.211553 21426 tablet_bootstrap.cc:654] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:24.212090 21426 log.cc:824] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:24.212734 21426 tablet_bootstrap.cc:492] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: No bootstrap required, opened a new log
I20260928 17:50:24.212797 21426 ts_tablet_manager.cc:1397] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:50:24.213443 21427 raft_consensus.cc:348] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } }
I20260928 17:50:24.213402 21425 raft_consensus.cc:348] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } }
I20260928 17:50:24.213586 21425 raft_consensus.cc:374] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:24.213586 21427 raft_consensus.cc:374] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:24.213619 21427 raft_consensus.cc:729] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6475b9ac70de429285c9859b1daa0f4c, State: Initialized, Role: FOLLOWER
I20260928 17:50:24.213620 21425 raft_consensus.cc:729] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983bd752223e49e092368be390489d4f, State: Initialized, Role: FOLLOWER
I20260928 17:50:24.213754 21427 consensus_queue.cc:260] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } }
I20260928 17:50:24.213754 21425 consensus_queue.cc:260] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } }
I20260928 17:50:24.214012 21425 ts_tablet_manager.cc:1428] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 17:50:24.214012 21427 ts_tablet_manager.cc:1428] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 17:50:24.214136 21143 heartbeater.cc:503] Master 127.3.222.62:41813 was elected leader, sending a full tablet report...
I20260928 17:50:24.214159 21277 heartbeater.cc:503] Master 127.3.222.62:41813 was elected leader, sending a full tablet report...
I20260928 17:50:24.214717 21426 raft_consensus.cc:348] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } }
I20260928 17:50:24.214886 21426 raft_consensus.cc:374] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:24.214921 21426 raft_consensus.cc:729] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee750e1da2a4503a54490274082db9f, State: Initialized, Role: FOLLOWER
I20260928 17:50:24.215025 21426 consensus_queue.cc:260] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } }
I20260928 17:50:24.215310 21411 heartbeater.cc:503] Master 127.3.222.62:41813 was elected leader, sending a full tablet report...
I20260928 17:50:24.215516 21426 ts_tablet_manager.cc:1428] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260928 17:50:24.283298 21278 tablet.cc:2369] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:50:24.330464 21432 raft_consensus.cc:482] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:50:24.330585 21432 raft_consensus.cc:504] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } }
I20260928 17:50:24.330905 21432 leader_election.cc:288] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eee750e1da2a4503a54490274082db9f (127.3.222.3:35409), 983bd752223e49e092368be390489d4f (127.3.222.1:41389)
I20260928 17:50:24.333513 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.331067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:41389 (local address 127.3.222.2:41963)
0928 17:50:24.331123 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.331135 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.331180 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.331383 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.331384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.331386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:24.331455 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.331457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.332040 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.332041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.332763 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.332765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.333282 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.333283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.333358 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.333362 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.333401 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.333440 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:50:24.333519 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.331155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:41963 (local address 127.3.222.1:41389)
0928 17:50:24.331249 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.331251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.331272 (+ 21us) server_negotiation.cc:407] Connection header received
0928 17:50:24.331303 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.331304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.331314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.331368 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:24.331510 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.331975 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.332848 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.333218 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.333277 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.333287 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.333423 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.333459 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 17:50:24.333782 21421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.331191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:45705 (local address 127.3.222.3:35409)
0928 17:50:24.331267 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.331269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.331504 (+ 235us) server_negotiation.cc:407] Connection header received
0928 17:50:24.331537 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.331538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.331550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.331603 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:24.331733 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.332184 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.333091 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.333460 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.333517 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.333527 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.333675 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.333722 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 17:50:24.333791 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.331073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:35409 (local address 127.3.222.2:45705)
0928 17:50:24.331435 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.331444 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.331494 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.331605 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.331606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.331607 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:24.331678 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.331679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.332253 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.332254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.333007 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.333009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.333529 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.333530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.333597 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.333602 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.333666 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.333703 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":210,"thread_start_us":93,"threads_started":1}
I20260928 17:50:24.333860 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9638a611c85481e84202d96b40ec3ce" candidate_uuid: "6475b9ac70de429285c9859b1daa0f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983bd752223e49e092368be390489d4f" is_pre_election: true
I20260928 17:50:24.334012 21097 raft_consensus.cc:2454] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6475b9ac70de429285c9859b1daa0f4c in term 0.
I20260928 17:50:24.334168 21365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9638a611c85481e84202d96b40ec3ce" candidate_uuid: "6475b9ac70de429285c9859b1daa0f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee750e1da2a4503a54490274082db9f" is_pre_election: true
I20260928 17:50:24.334229 21167 leader_election.cc:302] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6475b9ac70de429285c9859b1daa0f4c, 983bd752223e49e092368be390489d4f; no voters:
I20260928 17:50:24.334302 21365 raft_consensus.cc:2454] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6475b9ac70de429285c9859b1daa0f4c in term 0.
I20260928 17:50:24.334374 21432 raft_consensus.cc:2781] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:50:24.334440 21432 raft_consensus.cc:482] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:50:24.334472 21432 raft_consensus.cc:3037] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:24.335127 21432 raft_consensus.cc:504] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } }
I20260928 17:50:24.335263 21432 leader_election.cc:288] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [CANDIDATE]: Term 1 election: Requested vote from peers eee750e1da2a4503a54490274082db9f (127.3.222.3:35409), 983bd752223e49e092368be390489d4f (127.3.222.1:41389)
I20260928 17:50:24.335408 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9638a611c85481e84202d96b40ec3ce" candidate_uuid: "6475b9ac70de429285c9859b1daa0f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983bd752223e49e092368be390489d4f"
I20260928 17:50:24.335457 21365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9638a611c85481e84202d96b40ec3ce" candidate_uuid: "6475b9ac70de429285c9859b1daa0f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee750e1da2a4503a54490274082db9f"
I20260928 17:50:24.335481 21097 raft_consensus.cc:3037] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:24.335520 21365 raft_consensus.cc:3037] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:24.336184 21097 raft_consensus.cc:2454] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6475b9ac70de429285c9859b1daa0f4c in term 1.
I20260928 17:50:24.336192 21365 raft_consensus.cc:2454] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6475b9ac70de429285c9859b1daa0f4c in term 1.
I20260928 17:50:24.336344 21167 leader_election.cc:302] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6475b9ac70de429285c9859b1daa0f4c, 983bd752223e49e092368be390489d4f; no voters:
I20260928 17:50:24.336431 21432 raft_consensus.cc:2781] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:24.336577 21432 raft_consensus.cc:686] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 1 LEADER]: Becoming Leader. State: Replica: 6475b9ac70de429285c9859b1daa0f4c, State: Running, Role: LEADER
I20260928 17:50:24.336668 21432 consensus_queue.cc:237] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } }
I20260928 17:50:24.337479 20950 catalog_manager.cc:5495] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6475b9ac70de429285c9859b1daa0f4c (127.3.222.2). New cstate: current_term: 1 leader_uuid: "6475b9ac70de429285c9859b1daa0f4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6475b9ac70de429285c9859b1daa0f4c" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 } health_report { overall_health: UNKNOWN } } }
I20260928 17:50:24.352883 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.350510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.3.222.2:33117)
0928 17:50:24.350563 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.350565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.350588 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:50:24.350681 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.350682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.350707 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.350761 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:24.350866 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.351394 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.352162 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.352359 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.352413 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.352556 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:24.352661 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:24.352731 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.352766 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.352787 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:50:24.354615 21097 raft_consensus.cc:1261] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [term 1 FOLLOWER]: Refusing update from remote peer 6475b9ac70de429285c9859b1daa0f4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:24.354615 21365 raft_consensus.cc:1261] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [term 1 FOLLOWER]: Refusing update from remote peer 6475b9ac70de429285c9859b1daa0f4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:24.354955 21435 consensus_queue.cc:1035] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "983bd752223e49e092368be390489d4f" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 41389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:24.355083 21432 consensus_queue.cc:1035] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee750e1da2a4503a54490274082db9f" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:24.357213 21438 mvcc.cc:205] Tried to move back new op lower bound from 7334370608553865216 to 7334370608483512320. Current Snapshot: MvccSnapshot[applied={T|T < 7334370608553865216}]
I20260928 17:50:24.358340 21440 mvcc.cc:205] Tried to move back new op lower bound from 7334370608553865216 to 7334370608483512320. Current Snapshot: MvccSnapshot[applied={T|T < 7334370608553865216}]
I20260928 17:50:24.359565 20951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:24.359648 20951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:24.360370 20951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b451ec2da0de489c93888a83f684609c: Sending DeleteTablet for 3 replicas of tablet a9638a611c85481e84202d96b40ec3ce
I20260928 17:50:24.360646 21211 tablet_service.cc:1514] Processing DeleteTablet for tablet a9638a611c85481e84202d96b40ec3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:24 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:58042
I20260928 17:50:24.360679 21345 tablet_service.cc:1514] Processing DeleteTablet for tablet a9638a611c85481e84202d96b40ec3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:24 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:45522
I20260928 17:50:24.360687 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet a9638a611c85481e84202d96b40ec3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:24 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:47522
I20260928 17:50:24.360930 21450 tablet_replica.cc:330] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f: stopping tablet replica
I20260928 17:50:24.361032 21450 raft_consensus.cc:2229] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:24.361132 21450 raft_consensus.cc:2258] T a9638a611c85481e84202d96b40ec3ce P 983bd752223e49e092368be390489d4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:24.361202 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 21014
I20260928 17:50:24.362612 21449 tablet_replica.cc:330] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: stopping tablet replica
I20260928 17:50:24.362718 21449 raft_consensus.cc:2229] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:24.362818 21449 raft_consensus.cc:2258] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:24.363082 21448 tablet_replica.cc:330] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: stopping tablet replica
I20260928 17:50:24.363178 21448 raft_consensus.cc:2229] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:50:24.363289 21448 raft_consensus.cc:2258] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:24.363392 21449 ts_tablet_manager.cc:1905] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:50:24.363620 21448 ts_tablet_manager.cc:1905] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:50:24.365830 21449 ts_tablet_manager.cc:1918] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:24.365829 21448 ts_tablet_manager.cc:1918] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:24.365901 21448 log.cc:1196] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/a9638a611c85481e84202d96b40ec3ce
I20260928 17:50:24.366007 21449 log.cc:1196] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/a9638a611c85481e84202d96b40ec3ce
I20260928 17:50:24.366169 21448 ts_tablet_manager.cc:1939] T a9638a611c85481e84202d96b40ec3ce P 6475b9ac70de429285c9859b1daa0f4c: Deleting consensus metadata
I20260928 17:50:24.366369 21449 ts_tablet_manager.cc:1939] T a9638a611c85481e84202d96b40ec3ce P eee750e1da2a4503a54490274082db9f: Deleting consensus metadata
I20260928 17:50:24.366562 20939 catalog_manager.cc:4841] TS 6475b9ac70de429285c9859b1daa0f4c (127.3.222.2:33117): tablet a9638a611c85481e84202d96b40ec3ce (table test-table [id=5430c51ed0ad45da86c78d116af1c2f4]) successfully deleted
I20260928 17:50:24.366820 20936 catalog_manager.cc:4841] TS eee750e1da2a4503a54490274082db9f (127.3.222.3:35409): tablet a9638a611c85481e84202d96b40ec3ce (table test-table [id=5430c51ed0ad45da86c78d116af1c2f4]) successfully deleted
W20260928 17:50:24.367511 20939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:41389 (error 108)
I20260928 17:50:24.367681 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 21148
I20260928 17:50:24.368319 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.367808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:41389 (local address 127.0.0.1:47530)
0928 17:50:24.368259 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.368291 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:41389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434}
W20260928 17:50:24.368453 20939 catalog_manager.cc:4584] TS 983bd752223e49e092368be390489d4f (127.3.222.1:41389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9638a611c85481e84202d96b40ec3ce: Network error: Client connection negotiation failed: client connection to 127.3.222.1:41389: connect: Connection refused (error 111)
I20260928 17:50:24.372666 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 21282
I20260928 17:50:24.376469 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 20920
2026-09-28T17:50:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1860 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T17:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:24Z Disabled control of system clock
I20260928 17:50:24.401772 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41367
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:24.478487 21455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:24.478709 21455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:24.478729 21455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:24.479899 21455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:24.479946 21455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:24.479959 21455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:24.479969 21455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:24.479979 21455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:24.481333 21455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:24.481559 21455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:24.481848 21455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:24.486640 21461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.486760 21455 server_base.cc:1034] running on GCE node
W20260928 17:50:24.486654 21463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:24.486654 21460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.487133 21455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:24.487427 21455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:24.488580 21455 hybrid_clock.cc:648] HybridClock initialized: now 1790617824488567 us; error 27 us; skew 500 ppm
I20260928 17:50:24.489786 21455 webserver.cc:460] Webserver started at http://127.3.222.62:36613/ using document root <none> and password file <none>
I20260928 17:50:24.489996 21455 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:24.490038 21455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:24.490151 21455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:24.491190 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "ade8f1e59bf94ca18862c5ed02b2fb45"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "6eccdf13dfb04026bc350a11c3831eab"
server_key_iv: "e119abef4ca0b0ea7c2e7c8a4f6792eb"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.491557 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "ade8f1e59bf94ca18862c5ed02b2fb45"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "6eccdf13dfb04026bc350a11c3831eab"
server_key_iv: "e119abef4ca0b0ea7c2e7c8a4f6792eb"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.492799 21455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:24.493474 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.493640 21455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:24.493752 21455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "ade8f1e59bf94ca18862c5ed02b2fb45"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "6eccdf13dfb04026bc350a11c3831eab"
server_key_iv: "e119abef4ca0b0ea7c2e7c8a4f6792eb"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.493813 21455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:24.516390 21455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:24.516687 21455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:24.516794 21455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:24.520316 21455 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:41367
I20260928 17:50:24.520360 21521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:41367 every 8 connection(s)
I20260928 17:50:24.520654 21455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:24.521584 21522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:24.523849 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45: Bootstrap starting.
I20260928 17:50:24.524402 21522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:24.524654 21522 log.cc:824] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:24.525252 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45: No bootstrap required, opened a new log
I20260928 17:50:24.525601 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 21455
I20260928 17:50:24.525712 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:24.525907 3960 external_mini_cluster.cc:1428] Setting key 44e6f539f59a6a0c961f203be9a93481
I20260928 17:50:24.527277 21522 raft_consensus.cc:348] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade8f1e59bf94ca18862c5ed02b2fb45" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41367 } }
I20260928 17:50:24.527426 21522 raft_consensus.cc:374] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:24.527459 21522 raft_consensus.cc:729] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade8f1e59bf94ca18862c5ed02b2fb45, State: Initialized, Role: FOLLOWER
I20260928 17:50:24.527582 21522 consensus_queue.cc:260] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade8f1e59bf94ca18862c5ed02b2fb45" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41367 } }
I20260928 17:50:24.527647 21522 raft_consensus.cc:388] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:24.527676 21522 raft_consensus.cc:482] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:24.527720 21522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:24.528188 21522 raft_consensus.cc:504] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade8f1e59bf94ca18862c5ed02b2fb45" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41367 } }
I20260928 17:50:24.528275 21522 leader_election.cc:302] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ade8f1e59bf94ca18862c5ed02b2fb45; no voters:
I20260928 17:50:24.528429 21522 leader_election.cc:288] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:24.528457 21527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:24.528553 21527 raft_consensus.cc:686] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [term 1 LEADER]: Becoming Leader. State: Replica: ade8f1e59bf94ca18862c5ed02b2fb45, State: Running, Role: LEADER
I20260928 17:50:24.528631 21527 consensus_queue.cc:237] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade8f1e59bf94ca18862c5ed02b2fb45" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41367 } }
I20260928 17:50:24.528666 21522 sys_catalog.cc:564] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:24.528935 21527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ade8f1e59bf94ca18862c5ed02b2fb45. Latest consensus state: current_term: 1 leader_uuid: "ade8f1e59bf94ca18862c5ed02b2fb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade8f1e59bf94ca18862c5ed02b2fb45" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41367 } } }
I20260928 17:50:24.529023 21527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:24.529165 21528 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ade8f1e59bf94ca18862c5ed02b2fb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade8f1e59bf94ca18862c5ed02b2fb45" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41367 } } }
I20260928 17:50:24.529232 21528 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45 [sys.catalog]: This master's current role is: LEADER
W20260928 17:50:24.530215 21543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:24.530267 21543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:24.530308 21533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:24.530452 21533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:24.531848 21533 catalog_manager.cc:1347] Generated new cluster ID: dd5a2c4500254994803003fb59f9b87e
I20260928 17:50:24.531894 21533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:24.532261 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.526406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.3.222.62:41367)
0928 17:50:24.526606 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.526614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.526637 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:50:24.527197 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.527204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.527301 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.527431 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:24.527634 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.529100 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.530212 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.530721 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.530829 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.531566 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:24.531569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:24.531569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:24.531767 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:24.531819 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:24.531834 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:24.531900 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.531972 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.532049 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260928 17:50:24.537550 21533 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:24.538270 21533 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:24.547079 21533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45: Generated new TSK 0
I20260928 17:50:24.547261 21533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:24.549125 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41367
--builtin_ntp_servers=127.3.222.20:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:24.627713 21546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:24.627903 21546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:24.627924 21546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:24.629076 21546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:24.629122 21546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:24.629156 21546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:50:24.630553 21546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:24.630779 21546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:24.631054 21546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:24.635691 21551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:24.635727 21554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.635846 21546 server_base.cc:1034] running on GCE node
W20260928 17:50:24.635735 21552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.636183 21546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:24.636458 21546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:24.637629 21546 hybrid_clock.cc:648] HybridClock initialized: now 1790617824637606 us; error 37 us; skew 500 ppm
I20260928 17:50:24.639034 21546 webserver.cc:460] Webserver started at http://127.3.222.1:40561/ using document root <none> and password file <none>
I20260928 17:50:24.639312 21546 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:24.639369 21546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:24.639485 21546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:24.640810 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "950ec07883cd4fb8b0f4c962d94d74ba"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "b48c0b30b6b105609969a955dd7c646e"
server_key_iv: "67e709968135fb34609eb1fef9959d4f"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.641166 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "950ec07883cd4fb8b0f4c962d94d74ba"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "b48c0b30b6b105609969a955dd7c646e"
server_key_iv: "67e709968135fb34609eb1fef9959d4f"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.642657 21546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:50:24.643527 21560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.643707 21546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:50:24.643792 21546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "950ec07883cd4fb8b0f4c962d94d74ba"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "b48c0b30b6b105609969a955dd7c646e"
server_key_iv: "67e709968135fb34609eb1fef9959d4f"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.643862 21546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:24.667016 21546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:24.667341 21546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:24.667469 21546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:24.667676 21546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:24.668006 21546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:24.668040 21546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.668068 21546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:24.668092 21546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.673350 21546 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:44377
I20260928 17:50:24.673412 21673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:44377 every 8 connection(s)
I20260928 17:50:24.673764 21546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:50:24.673894 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 21546
I20260928 17:50:24.674007 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:50:24.674232 3960 external_mini_cluster.cc:1428] Setting key 9ea6211a9c9b2f4ab343837ff7564e44
I20260928 17:50:24.675129 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41367
--builtin_ntp_servers=127.3.222.20:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 17:50:24.678946 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.674487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:58595 (local address 127.3.222.62:41367)
0928 17:50:24.674547 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.674550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.674806 (+ 256us) server_negotiation.cc:407] Connection header received
0928 17:50:24.675388 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.675389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.675400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.675461 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:24.676122 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.676607 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.677555 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.677754 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.677815 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.678690 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:24.678691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:24.678692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:24.678696 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:24.678710 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:24.678714 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:24.678753 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.678850 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.678885 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:50:24.679227 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.674480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41367 (local address 127.3.222.1:58595)
0928 17:50:24.674673 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.674715 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.675290 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.675536 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.675540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.675551 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:24.676037 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.676042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.676684 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.676684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.677474 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.677477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.677852 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.677853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.678023 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.678537 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:50:24.678538 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:24.678578 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:50:24.678606 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:24.678772 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:24.678777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:24.678781 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.678851 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.678967 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260928 17:50:24.679705 21674 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41367
I20260928 17:50:24.679801 21674 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:24.680009 21674 heartbeater.cc:511] Master 127.3.222.62:41367 requested a full tablet report, sending...
I20260928 17:50:24.680516 21486 ts_manager.cc:194] Registered new tserver with Master: 950ec07883cd4fb8b0f4c962d94d74ba (127.3.222.1:44377)
I20260928 17:50:24.681095 21486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.1:58595
W20260928 17:50:24.753023 21677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:24.753211 21677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:24.753229 21677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:24.754351 21677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:24.754390 21677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:24.754413 21677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:50:24.755736 21677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:24.755934 21677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:24.756191 21677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:24.760797 21682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:24.760839 21685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.760836 21677 server_base.cc:1034] running on GCE node
W20260928 17:50:24.760788 21683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.761134 21677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:24.761355 21677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:24.762497 21677 hybrid_clock.cc:648] HybridClock initialized: now 1790617824762483 us; error 27 us; skew 500 ppm
I20260928 17:50:24.763625 21677 webserver.cc:460] Webserver started at http://127.3.222.2:39759/ using document root <none> and password file <none>
I20260928 17:50:24.763825 21677 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:24.763871 21677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:24.764001 21677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:24.765012 21677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "1bf6df9fa51b4cc9a3ee1be0249df248"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "86381c12db839ce47399f9de52611e8b"
server_key_iv: "bcb4e6b6cfddf9e2f74dd7d3a6957262"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.765337 21677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "1bf6df9fa51b4cc9a3ee1be0249df248"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "86381c12db839ce47399f9de52611e8b"
server_key_iv: "bcb4e6b6cfddf9e2f74dd7d3a6957262"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.766459 21677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:24.767037 21691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.767186 21677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:24.767292 21677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "1bf6df9fa51b4cc9a3ee1be0249df248"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "86381c12db839ce47399f9de52611e8b"
server_key_iv: "bcb4e6b6cfddf9e2f74dd7d3a6957262"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.767356 21677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:24.774786 21677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:24.775038 21677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:24.775156 21677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:24.775370 21677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:24.775718 21677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:24.775756 21677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.775784 21677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:24.775808 21677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.781118 21677 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:42259
I20260928 17:50:24.781163 21804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:42259 every 8 connection(s)
I20260928 17:50:24.781458 21677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:50:24.786463 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.782117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:36791 (local address 127.3.222.62:41367)
0928 17:50:24.782203 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.782205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.782406 (+ 201us) server_negotiation.cc:407] Connection header received
0928 17:50:24.782974 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.782976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.782988 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.783040 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:24.783748 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.784241 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.785195 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.785379 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.785433 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.786209 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:24.786210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:24.786211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:24.786215 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:24.786229 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:24.786233 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:24.786274 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.786355 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.786381 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 17:50:24.786657 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.782087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41367 (local address 127.3.222.2:36791)
0928 17:50:24.782290 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.782328 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.782875 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.783108 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.783112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.783120 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:24.783658 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.783663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.784325 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.784326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.785107 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.785109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.785450 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.785451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.785571 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.786066 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:50:24.786066 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:24.786097 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:50:24.786119 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:24.786282 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:24.786287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:24.786290 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.786352 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.786454 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260928 17:50:24.787052 21805 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41367
I20260928 17:50:24.787135 21805 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:24.787339 21805 heartbeater.cc:511] Master 127.3.222.62:41367 requested a full tablet report, sending...
I20260928 17:50:24.787679 21486 ts_manager.cc:194] Registered new tserver with Master: 1bf6df9fa51b4cc9a3ee1be0249df248 (127.3.222.2:42259)
I20260928 17:50:24.788038 21486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.2:36791
I20260928 17:50:24.789399 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 21677
I20260928 17:50:24.789476 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:50:24.789647 3960 external_mini_cluster.cc:1428] Setting key ac123638f1a9b6ce59b3d3f4784b34a1
I20260928 17:50:24.790417 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41367
--builtin_ntp_servers=127.3.222.20:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:24.868913 21808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:24.869172 21808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:24.869204 21808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:24.870815 21808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:24.870863 21808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:24.870895 21808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:50:24.872704 21808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:24.872965 21808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:24.873291 21808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:24.879534 21816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:24.879559 21814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.879896 21808 server_base.cc:1034] running on GCE node
W20260928 17:50:24.879535 21813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:24.880152 21808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:24.880448 21808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:24.881593 21808 hybrid_clock.cc:648] HybridClock initialized: now 1790617824881574 us; error 28 us; skew 500 ppm
I20260928 17:50:24.883001 21808 webserver.cc:460] Webserver started at http://127.3.222.3:37753/ using document root <none> and password file <none>
I20260928 17:50:24.883445 21808 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:24.883514 21808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:24.883647 21808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:24.885108 21808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "e04574e6013041699f6d85910cb0eca9"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "5eea6ca6d35ab52da002b8c5e06e280e"
server_key_iv: "130c7d0ae5b09b41c9c591548499711b"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.885509 21808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "e04574e6013041699f6d85910cb0eca9"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "5eea6ca6d35ab52da002b8c5e06e280e"
server_key_iv: "130c7d0ae5b09b41c9c591548499711b"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.887147 21808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 17:50:24.887953 21822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.888104 21808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:50:24.888195 21808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "e04574e6013041699f6d85910cb0eca9"
format_stamp: "Formatted at 2026-09-28 17:50:24 on dist-test-slave-w3bm"
server_key: "5eea6ca6d35ab52da002b8c5e06e280e"
server_key_iv: "130c7d0ae5b09b41c9c591548499711b"
server_key_version: "encryptionkey@0"
I20260928 17:50:24.888270 21808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:24.913496 21808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:24.913815 21808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:24.913938 21808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:24.914194 21808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:24.914511 21808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:24.914552 21808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.914582 21808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:24.914606 21808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:24.919996 21808 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:46287
I20260928 17:50:24.920047 21935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:46287 every 8 connection(s)
I20260928 17:50:24.920342 21808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:50:24.924839 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 21808
I20260928 17:50:24.924929 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:50:24.925110 3960 external_mini_cluster.cc:1428] Setting key 74c0468cf9709f078a2892efca440224
I20260928 17:50:24.925196 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.920985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:44527 (local address 127.3.222.62:41367)
0928 17:50:24.921062 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.921064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.921290 (+ 226us) server_negotiation.cc:407] Connection header received
0928 17:50:24.921744 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.921745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.921756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.921812 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:24.922472 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.922962 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.923971 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.924156 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.924233 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.924951 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:24.924953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:24.924953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:24.924958 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:24.924975 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:24.924979 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:24.925025 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.925092 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.925118 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:50:24.925418 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.920975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41367 (local address 127.3.222.3:44527)
0928 17:50:24.921147 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.921210 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.921673 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.921881 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.921884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.921892 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:24.922403 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.922408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.923041 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.923042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.923897 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.923900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.924243 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.924244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.924353 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.924819 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:50:24.924819 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:24.924848 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:50:24.924870 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:24.925025 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:24.925031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:24.925034 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.925098 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.925206 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260928 17:50:24.925910 21936 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41367
I20260928 17:50:24.926000 21936 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:24.926142 21936 heartbeater.cc:511] Master 127.3.222.62:41367 requested a full tablet report, sending...
I20260928 17:50:24.926472 21486 ts_manager.cc:194] Registered new tserver with Master: e04574e6013041699f6d85910cb0eca9 (127.3.222.3:46287)
I20260928 17:50:24.926819 21486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.3:44527
I20260928 17:50:24.936661 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:50:24.940513 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.937926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.3.222.62:41367)
0928 17:50:24.937969 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.937971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.938068 (+ 97us) server_negotiation.cc:407] Connection header received
0928 17:50:24.938110 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.938110 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.938119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.938164 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:24.938321 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.938817 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.939563 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.939771 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.939821 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.940343 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:24.940344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:24.940344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:24.940348 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:24.940360 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:24.940363 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:24.940400 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.940437 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.940456 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:50:24.942028 21486 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:50:24.943715 21486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:50:24.949468 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.945810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:46287 (local address 127.0.0.1:35098)
0928 17:50:24.946017 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.946058 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.946122 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.946356 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.946359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.946361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:24.946421 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.946422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.947553 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.947554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.948354 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.948356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.949172 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.949173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.949256 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.949270 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.949340 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.949390 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260928 17:50:24.949473 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.946046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.3.222.3:46287)
0928 17:50:24.946183 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.946187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.946208 (+ 21us) server_negotiation.cc:407] Connection header received
0928 17:50:24.946246 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.946248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.946263 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.946342 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:24.946490 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.947497 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.948433 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.949083 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.949152 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.949252 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.949350 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.949389 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260928 17:50:24.950186 21870 tablet_service.cc:1467] Processing CreateTablet for tablet d3312bd2343b43fc84a76bf296ded614 (DEFAULT_TABLE table=test-table [id=b1f9b254a59b4ad1aab2d79a120eb3b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:24.950501 21870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3312bd2343b43fc84a76bf296ded614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:24.951018 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.945810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:42259 (local address 127.0.0.1:42386)
0928 17:50:24.946254 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.946263 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.946301 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.947508 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.947509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.947510 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:24.947538 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.947539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.948919 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.948921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.949949 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.949952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.950795 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.950796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.950856 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.950860 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.950911 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.950948 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260928 17:50:24.951033 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.947049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.3.222.2:42259)
0928 17:50:24.947361 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.947364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.947373 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:50:24.947403 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.947405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.947416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.947612 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:24.947676 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.948848 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.950030 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.950713 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.950771 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.950849 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.950924 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.950958 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":93,"threads_started":1}
I20260928 17:50:24.951339 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.947108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.3.222.1:44377)
0928 17:50:24.947306 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:24.947310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:24.947320 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:50:24.947357 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:24.947359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:24.947373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:24.947450 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:24.947598 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.948714 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.949848 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.950489 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.950562 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.950643 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:24.951188 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 17:50:24.951254 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":118,"threads_started":1}
I20260928 17:50:24.951581 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:24.945810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44377 (local address 127.0.0.1:44060)
0928 17:50:24.947135 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:24.947145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:24.947194 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:24.947459 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:24.947459 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:24.947462 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:24.947506 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.947507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.948790 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.948791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.949763 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:24.949766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:24.950632 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:24.950633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:24.950691 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:24.950695 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:24.950763 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:50:24.950799 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"mutex_wait_us":10,"thread_start_us":110,"threads_started":1}
I20260928 17:50:24.951622 21739 tablet_service.cc:1467] Processing CreateTablet for tablet d3312bd2343b43fc84a76bf296ded614 (DEFAULT_TABLE table=test-table [id=b1f9b254a59b4ad1aab2d79a120eb3b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:24.951885 21739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3312bd2343b43fc84a76bf296ded614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:24.953295 21950 tablet_bootstrap.cc:492] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: Bootstrap starting.
I20260928 17:50:24.953980 21950 tablet_bootstrap.cc:654] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:24.954288 21950 log.cc:824] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:24.954521 21951 tablet_bootstrap.cc:492] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: Bootstrap starting.
I20260928 17:50:24.954999 21950 tablet_bootstrap.cc:492] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: No bootstrap required, opened a new log
I20260928 17:50:24.955070 21950 ts_tablet_manager.cc:1397] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:24.955137 21951 tablet_bootstrap.cc:654] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:24.955029 21608 tablet_service.cc:1467] Processing CreateTablet for tablet d3312bd2343b43fc84a76bf296ded614 (DEFAULT_TABLE table=test-table [id=b1f9b254a59b4ad1aab2d79a120eb3b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:24.955374 21608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3312bd2343b43fc84a76bf296ded614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:24.955416 21951 log.cc:824] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:24.955984 21951 tablet_bootstrap.cc:492] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: No bootstrap required, opened a new log
I20260928 17:50:24.956037 21951 ts_tablet_manager.cc:1397] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:24.957479 21950 raft_consensus.cc:348] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } }
I20260928 17:50:24.957641 21950 raft_consensus.cc:374] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:24.957692 21950 raft_consensus.cc:729] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04574e6013041699f6d85910cb0eca9, State: Initialized, Role: FOLLOWER
I20260928 17:50:24.957820 21950 consensus_queue.cc:260] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } }
I20260928 17:50:24.958108 21936 heartbeater.cc:503] Master 127.3.222.62:41367 was elected leader, sending a full tablet report...
I20260928 17:50:24.958003 21951 raft_consensus.cc:348] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } }
I20260928 17:50:24.958153 21951 raft_consensus.cc:374] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:24.958189 21951 raft_consensus.cc:729] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf6df9fa51b4cc9a3ee1be0249df248, State: Initialized, Role: FOLLOWER
I20260928 17:50:24.958293 21951 consensus_queue.cc:260] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } }
I20260928 17:50:24.958334 21950 ts_tablet_manager.cc:1428] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:50:24.958524 21951 ts_tablet_manager.cc:1428] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:50:24.958631 21805 heartbeater.cc:503] Master 127.3.222.62:41367 was elected leader, sending a full tablet report...
I20260928 17:50:24.958799 21955 tablet_bootstrap.cc:492] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba: Bootstrap starting.
I20260928 17:50:24.959410 21955 tablet_bootstrap.cc:654] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:24.959713 21955 log.cc:824] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:24.960309 21955 tablet_bootstrap.cc:492] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba: No bootstrap required, opened a new log
I20260928 17:50:24.960357 21955 ts_tablet_manager.cc:1397] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:24.962132 21955 raft_consensus.cc:348] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } }
I20260928 17:50:24.962261 21955 raft_consensus.cc:374] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:24.962282 21955 raft_consensus.cc:729] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950ec07883cd4fb8b0f4c962d94d74ba, State: Initialized, Role: FOLLOWER
I20260928 17:50:24.962371 21955 consensus_queue.cc:260] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } }
I20260928 17:50:24.962577 21955 ts_tablet_manager.cc:1428] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:50:24.962633 21674 heartbeater.cc:503] Master 127.3.222.62:41367 was elected leader, sending a full tablet report...
W20260928 17:50:25.031961 21806 tablet.cc:2369] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:50:25.047770 21956 raft_consensus.cc:482] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:50:25.047911 21956 raft_consensus.cc:504] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } }
I20260928 17:50:25.048259 21956 leader_election.cc:288] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 950ec07883cd4fb8b0f4c962d94d74ba (127.3.222.1:44377), e04574e6013041699f6d85910cb0eca9 (127.3.222.3:46287)
I20260928 17:50:25.051002 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:25.048494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:36943 (local address 127.3.222.1:44377)
0928 17:50:25.048541 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:25.048543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:25.048566 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:50:25.048650 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:25.048651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:25.048664 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:25.048721 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:25.048874 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.049375 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.050343 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.050696 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.050760 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:25.050769 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:25.050905 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 17:50:25.050936 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:50:25.051004 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:25.048576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:41467 (local address 127.3.222.3:46287)
0928 17:50:25.048635 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:25.048637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:25.048662 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:50:25.048739 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:25.048740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:25.048749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:25.048805 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:25.048950 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.049415 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.050355 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.050701 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.050760 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:25.050770 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:25.050898 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 17:50:25.050935 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:50:25.051016 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:25.048430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44377 (local address 127.3.222.2:36943)
0928 17:50:25.048515 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:25.048527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:25.048582 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:25.048732 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:25.048732 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:25.048734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:25.048811 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:25.048813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.049448 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.049449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:25.050262 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:25.050265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.050770 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.050770 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:25.050844 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:25.050846 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:50:25.050899 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 17:50:25.050936 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":25}
I20260928 17:50:25.051023 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:25.048434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:46287 (local address 127.3.222.2:41467)
0928 17:50:25.048608 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:25.048617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:25.048676 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:25.048815 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:25.048816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:25.048817 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:25.048891 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:25.048893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.049485 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.049486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:25.050274 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:25.050276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.050769 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.050770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:25.050840 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:25.050843 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:25.050886 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:50:25.050929 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":13,"thread_start_us":87,"threads_started":1}
I20260928 17:50:25.051333 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3312bd2343b43fc84a76bf296ded614" candidate_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" is_pre_election: true
I20260928 17:50:25.051483 21628 raft_consensus.cc:2454] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf6df9fa51b4cc9a3ee1be0249df248 in term 0.
I20260928 17:50:25.051496 21890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3312bd2343b43fc84a76bf296ded614" candidate_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04574e6013041699f6d85910cb0eca9" is_pre_election: true
I20260928 17:50:25.051625 21890 raft_consensus.cc:2454] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf6df9fa51b4cc9a3ee1be0249df248 in term 0.
I20260928 17:50:25.051716 21693 leader_election.cc:302] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf6df9fa51b4cc9a3ee1be0249df248, 950ec07883cd4fb8b0f4c962d94d74ba; no voters:
I20260928 17:50:25.051834 21956 raft_consensus.cc:2781] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:50:25.051888 21956 raft_consensus.cc:482] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:50:25.051918 21956 raft_consensus.cc:3037] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:25.052588 21956 raft_consensus.cc:504] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } }
I20260928 17:50:25.052702 21956 leader_election.cc:288] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [CANDIDATE]: Term 1 election: Requested vote from peers 950ec07883cd4fb8b0f4c962d94d74ba (127.3.222.1:44377), e04574e6013041699f6d85910cb0eca9 (127.3.222.3:46287)
I20260928 17:50:25.052855 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3312bd2343b43fc84a76bf296ded614" candidate_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950ec07883cd4fb8b0f4c962d94d74ba"
I20260928 17:50:25.052896 21890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3312bd2343b43fc84a76bf296ded614" candidate_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04574e6013041699f6d85910cb0eca9"
I20260928 17:50:25.052959 21890 raft_consensus.cc:3037] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:25.052944 21628 raft_consensus.cc:3037] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:25.053820 21890 raft_consensus.cc:2454] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf6df9fa51b4cc9a3ee1be0249df248 in term 1.
I20260928 17:50:25.053820 21628 raft_consensus.cc:2454] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf6df9fa51b4cc9a3ee1be0249df248 in term 1.
I20260928 17:50:25.053985 21693 leader_election.cc:302] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf6df9fa51b4cc9a3ee1be0249df248, 950ec07883cd4fb8b0f4c962d94d74ba; no voters:
I20260928 17:50:25.054067 21956 raft_consensus.cc:2781] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:25.054208 21956 raft_consensus.cc:686] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 1 LEADER]: Becoming Leader. State: Replica: 1bf6df9fa51b4cc9a3ee1be0249df248, State: Running, Role: LEADER
I20260928 17:50:25.054293 21956 consensus_queue.cc:237] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } }
I20260928 17:50:25.055063 21485 catalog_manager.cc:5495] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bf6df9fa51b4cc9a3ee1be0249df248 (127.3.222.2). New cstate: current_term: 1 leader_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bf6df9fa51b4cc9a3ee1be0249df248" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 42259 } health_report { overall_health: HEALTHY } } }
I20260928 17:50:25.063552 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:25.061540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.3.222.2:42259)
0928 17:50:25.061593 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:25.061595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:25.061602 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:50:25.061629 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:25.061629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:25.061644 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:25.061709 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:25.061790 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.062240 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.062938 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:25.063123 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:25.063174 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:25.063307 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:25.063397 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:25.063456 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:25.063473 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 17:50:25.063487 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:50:25.065142 21628 raft_consensus.cc:1261] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba [term 1 FOLLOWER]: Refusing update from remote peer 1bf6df9fa51b4cc9a3ee1be0249df248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:25.065150 21890 raft_consensus.cc:1261] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [term 1 FOLLOWER]: Refusing update from remote peer 1bf6df9fa51b4cc9a3ee1be0249df248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:25.065483 21960 consensus_queue.cc:1035] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04574e6013041699f6d85910cb0eca9" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 46287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:25.065595 21956 consensus_queue.cc:1035] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "950ec07883cd4fb8b0f4c962d94d74ba" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:25.070367 21485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:25.070461 21485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:25.071167 21485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ade8f1e59bf94ca18862c5ed02b2fb45: Sending DeleteTablet for 3 replicas of tablet d3312bd2343b43fc84a76bf296ded614
I20260928 17:50:25.071483 21608 tablet_service.cc:1514] Processing DeleteTablet for tablet d3312bd2343b43fc84a76bf296ded614 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:25 UTC) from {username='slave'} at 127.0.0.1:44060
I20260928 17:50:25.071501 21870 tablet_service.cc:1514] Processing DeleteTablet for tablet d3312bd2343b43fc84a76bf296ded614 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:25 UTC) from {username='slave'} at 127.0.0.1:35098
I20260928 17:50:25.071540 21739 tablet_service.cc:1514] Processing DeleteTablet for tablet d3312bd2343b43fc84a76bf296ded614 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:25 UTC) from {username='slave'} at 127.0.0.1:42386
I20260928 17:50:25.071786 21973 tablet_replica.cc:330] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba: stopping tablet replica
I20260928 17:50:25.071895 21973 raft_consensus.cc:2229] T d3312bd2343b43fc84a76bf296ded614 P 950ec07883cd4fb8b0f4c962d94d74ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:25.071930 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 21546
I20260928 17:50:25.072628 21974 tablet_replica.cc:330] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: stopping tablet replica
I20260928 17:50:25.072723 21974 raft_consensus.cc:2229] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:25.072813 21974 raft_consensus.cc:2258] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:25.073211 21974 ts_tablet_manager.cc:1905] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:50:25.074753 21975 tablet_replica.cc:330] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: stopping tablet replica
I20260928 17:50:25.074851 21975 raft_consensus.cc:2229] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:50:25.074975 21975 raft_consensus.cc:2258] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:25.075099 21974 ts_tablet_manager.cc:1918] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:25.075160 21974 log.cc:1196] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/d3312bd2343b43fc84a76bf296ded614
I20260928 17:50:25.075416 21974 ts_tablet_manager.cc:1939] T d3312bd2343b43fc84a76bf296ded614 P e04574e6013041699f6d85910cb0eca9: Deleting consensus metadata
I20260928 17:50:25.075807 21470 catalog_manager.cc:4841] TS e04574e6013041699f6d85910cb0eca9 (127.3.222.3:46287): tablet d3312bd2343b43fc84a76bf296ded614 (table test-table [id=b1f9b254a59b4ad1aab2d79a120eb3b4]) successfully deleted
I20260928 17:50:25.076081 21975 ts_tablet_manager.cc:1905] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:50:25.077730 21975 ts_tablet_manager.cc:1918] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:25.077795 21975 log.cc:1196] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/d3312bd2343b43fc84a76bf296ded614
I20260928 17:50:25.078024 21975 ts_tablet_manager.cc:1939] T d3312bd2343b43fc84a76bf296ded614 P 1bf6df9fa51b4cc9a3ee1be0249df248: Deleting consensus metadata
I20260928 17:50:25.078374 21473 catalog_manager.cc:4841] TS 1bf6df9fa51b4cc9a3ee1be0249df248 (127.3.222.2:42259): tablet d3312bd2343b43fc84a76bf296ded614 (table test-table [id=b1f9b254a59b4ad1aab2d79a120eb3b4]) successfully deleted
W20260928 17:50:25.078604 21471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:44377 (error 108)
I20260928 17:50:25.079042 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:25.078940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:44377 (local address 127.0.0.1:44062)
0928 17:50:25.078974 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:25.079011 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 17:50:25.079146 21471 catalog_manager.cc:4584] TS 950ec07883cd4fb8b0f4c962d94d74ba (127.3.222.1:44377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3312bd2343b43fc84a76bf296ded614: Network error: Client connection negotiation failed: client connection to 127.3.222.1:44377: connect: Connection refused (error 111)
I20260928 17:50:25.079180 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 21677
I20260928 17:50:25.083448 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 21808
I20260928 17:50:25.086933 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 21455
2026-09-28T17:50:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (709 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:25 dist-test-slave-w3bm krb5kdc[21979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:25 dist-test-slave-w3bm krb5kdc[21979](info): set up 2 sockets
Sep 28 17:50:25 dist-test-slave-w3bm krb5kdc[21979](info): commencing operation
krb5kdc: starting...
W20260928 17:50:26.038705 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:26 dist-test-slave-w3bm krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:26.150983 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35735
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:35735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:26.228257 21992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:26.228447 21992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:26.228468 21992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:26.229568 21992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:26.229605 21992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:26.229619 21992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:26.229629 21992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:26.229640 21992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:26.231055 21992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:35735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:26.231274 21992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:26.231484 21992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:26.235908 21998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:26.235913 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:26.236078 21992 server_base.cc:1034] running on GCE node
W20260928 17:50:26.235916 21997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:26.236424 21992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:26.236671 21992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:26.237803 21992 hybrid_clock.cc:648] HybridClock initialized: now 1790617826237789 us; error 33 us; skew 500 ppm
Sep 28 17:50:26 dist-test-slave-w3bm krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:26.240161 21992 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:26.240607 21992 webserver.cc:460] Webserver started at http://127.3.222.62:34635/ using document root <none> and password file <none>
I20260928 17:50:26.240828 21992 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:26.240877 21992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:26.240998 21992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:26.242050 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "ab01ff388cdf49a39c1b332fda5bdd2f"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "22088335a17e9f610210601740ac40d3"
server_key_iv: "a6013688add3148dbe28f94321a00834"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.242368 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "ab01ff388cdf49a39c1b332fda5bdd2f"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "22088335a17e9f610210601740ac40d3"
server_key_iv: "a6013688add3148dbe28f94321a00834"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.243585 21992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:26.244328 22007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.244488 21992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:26.244566 21992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "ab01ff388cdf49a39c1b332fda5bdd2f"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "22088335a17e9f610210601740ac40d3"
server_key_iv: "a6013688add3148dbe28f94321a00834"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.244624 21992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:26.252487 21992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:26.254323 21992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:26.254446 21992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:26.258854 21992 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:35735
I20260928 17:50:26.259264 21992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:26.260421 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:26.263456 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f: Bootstrap starting.
I20260928 17:50:26.264117 22060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:26.264387 22060 log.cc:824] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:26.264652 22059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:35735 every 8 connection(s)
I20260928 17:50:26.265156 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f: No bootstrap required, opened a new log
I20260928 17:50:26.267422 22060 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01ff388cdf49a39c1b332fda5bdd2f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35735 } }
I20260928 17:50:26.267597 22060 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:26.267596 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 21992
I20260928 17:50:26.267639 22060 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab01ff388cdf49a39c1b332fda5bdd2f, State: Initialized, Role: FOLLOWER
I20260928 17:50:26.267699 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:26.267778 22060 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01ff388cdf49a39c1b332fda5bdd2f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35735 } }
I20260928 17:50:26.267853 22060 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:26.267879 22060 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:26.267891 3960 external_mini_cluster.cc:1428] Setting key 0822a91f8b54b54b283a4a3d6a866af9
I20260928 17:50:26.267920 22060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:26.268499 22060 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01ff388cdf49a39c1b332fda5bdd2f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35735 } }
I20260928 17:50:26.268599 22060 leader_election.cc:302] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab01ff388cdf49a39c1b332fda5bdd2f; no voters:
I20260928 17:50:26.268755 22060 leader_election.cc:288] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:26.268821 22065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:26.268987 22060 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:26.268993 22065 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [term 1 LEADER]: Becoming Leader. State: Replica: ab01ff388cdf49a39c1b332fda5bdd2f, State: Running, Role: LEADER
I20260928 17:50:26.269106 22065 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01ff388cdf49a39c1b332fda5bdd2f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35735 } }
I20260928 17:50:26.269634 22067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab01ff388cdf49a39c1b332fda5bdd2f. Latest consensus state: current_term: 1 leader_uuid: "ab01ff388cdf49a39c1b332fda5bdd2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01ff388cdf49a39c1b332fda5bdd2f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35735 } } }
I20260928 17:50:26.270006 22067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:26.270263 22080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:26.270808 22080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:26.271097 22066 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab01ff388cdf49a39c1b332fda5bdd2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01ff388cdf49a39c1b332fda5bdd2f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35735 } } }
I20260928 17:50:26.271183 22066 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:26.272271 22080 catalog_manager.cc:1347] Generated new cluster ID: 8040fc106ee342be921ab3f05ad6036a
I20260928 17:50:26.272321 22080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:50:26 dist-test-slave-w3bm krb5kdc[21979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:26.277675 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.268415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.3.222.62:35735)
0928 17:50:26.268586 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:26.268595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:26.268631 (+ 36us) server_negotiation.cc:407] Connection header received
0928 17:50:26.268746 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:26.268753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:26.268813 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:26.268953 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:26.269765 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.270996 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.272033 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.272490 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.272584 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.274710 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:26.274714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:26.274714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:26.274726 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:26.276353 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.276642 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.276645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.276646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.276740 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.276906 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.276907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.276907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.277127 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:26.277277 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:26.277354 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 17:50:26.277484 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260928 17:50:26.321729 22080 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:26.322301 22080 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:26.329557 22080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f: Generated new TSK 0
I20260928 17:50:26.329723 22080 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:50:26.363919 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35735
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:26.461884 22086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:26.462080 22086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:26.462100 22086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:26.463264 22086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:26.463304 22086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:26.463330 22086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:50:26.464771 22086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:26.464991 22086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:26.465194 22086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:26.469843 22092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:26.470013 22094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:26.470042 22091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:26.470088 22086 server_base.cc:1034] running on GCE node
I20260928 17:50:26.470312 22086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:26.470549 22086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:26.471679 22086 hybrid_clock.cc:648] HybridClock initialized: now 1790617826471671 us; error 25 us; skew 500 ppm
Sep 28 17:50:26 dist-test-slave-w3bm krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:26.473959 22086 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:50:26.474380 22086 webserver.cc:460] Webserver started at http://127.3.222.1:37541/ using document root <none> and password file <none>
I20260928 17:50:26.474586 22086 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:26.474632 22086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:26.474754 22086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:26.475816 22086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "2730d5fa671a4d1c951f1ea9d27c75f0"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "d9009fde7780f6799fcd6970c7aef799"
server_key_iv: "f361f2b1b8a681be8c9c21013de8ed50"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.476128 22086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "2730d5fa671a4d1c951f1ea9d27c75f0"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "d9009fde7780f6799fcd6970c7aef799"
server_key_iv: "f361f2b1b8a681be8c9c21013de8ed50"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.477357 22086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:50:26.478068 22101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.478252 22086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:50:26.478332 22086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "2730d5fa671a4d1c951f1ea9d27c75f0"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "d9009fde7780f6799fcd6970c7aef799"
server_key_iv: "f361f2b1b8a681be8c9c21013de8ed50"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.478394 22086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:26.491536 22086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:26.493280 22086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:26.493405 22086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:26.493597 22086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:26.493906 22086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:26.493938 22086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.493965 22086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:26.493981 22086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.499202 22086 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:46237
I20260928 17:50:26.499279 22214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:46237 every 8 connection(s)
I20260928 17:50:26.499564 22086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:50:26 dist-test-slave-w3bm krb5kdc[21979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:26.507925 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.500175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:42953 (local address 127.3.222.62:35735)
0928 17:50:26.500216 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:26.500218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:26.500394 (+ 176us) server_negotiation.cc:407] Connection header received
0928 17:50:26.500923 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:26.500925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:26.500942 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:26.500999 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:26.501886 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.502392 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.503371 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.503565 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.503618 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.505653 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:26.505654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:26.505655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:26.505662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:26.506808 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.507146 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.507147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.507147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.507198 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.507452 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.507452 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.507453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.507547 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:26.507592 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:26.507799 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 17:50:26.507861 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:50:26.508083 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.500142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35735 (local address 127.3.222.1:42953)
0928 17:50:26.500272 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:26.500312 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:26.500788 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:26.501068 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:26.501072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:26.501603 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:26.501786 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.501792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.502473 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.502474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.503282 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.503284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.503630 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.503631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.503753 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.504196 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:26.504203 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:26.505566 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:26.506878 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:26.506884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:26.506886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:26.507081 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:26.507281 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:26.507282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:26.507283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:26.507393 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:26.507592 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:26.507595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:26.507703 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 17:50:26.507779 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:50:26.507899 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260928 17:50:26.508535 22215 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35735
I20260928 17:50:26.508623 22215 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:26.508788 22215 heartbeater.cc:511] Master 127.3.222.62:35735 requested a full tablet report, sending...
I20260928 17:50:26.508991 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 22086
I20260928 17:50:26.509081 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:50:26.509291 3960 external_mini_cluster.cc:1428] Setting key f32ab5f45daadc53b5e7435aed84ddb3
I20260928 17:50:26.509379 22024 ts_manager.cc:194] Registered new tserver with Master: 2730d5fa671a4d1c951f1ea9d27c75f0 (127.3.222.1:46237)
I20260928 17:50:26.510244 22024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:42953
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:50:26.534338 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35735
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:26.612522 22220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:26.612718 22220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:26.612740 22220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:26.613879 22220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:26.613922 22220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:26.613947 22220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:50:26.615371 22220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:26.615576 22220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:26.615779 22220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:26.620486 22228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:26.620473 22225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:26.620530 22220 server_base.cc:1034] running on GCE node
W20260928 17:50:26.620466 22226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:26.620831 22220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:26.621083 22220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:26.622233 22220 hybrid_clock.cc:648] HybridClock initialized: now 1790617826622217 us; error 30 us; skew 500 ppm
Sep 28 17:50:26 dist-test-slave-w3bm krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:26.624599 22220 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:50:26.624990 22220 webserver.cc:460] Webserver started at http://127.3.222.2:35493/ using document root <none> and password file <none>
I20260928 17:50:26.625175 22220 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:26.625221 22220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:26.625340 22220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:26.626324 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "5ace053942374a7193e2100f472011da"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "3b66ade72362e14f5b68ba6b03f55200"
server_key_iv: "d2362573eb55b080a655e19c460a497d"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.626646 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "5ace053942374a7193e2100f472011da"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "3b66ade72362e14f5b68ba6b03f55200"
server_key_iv: "d2362573eb55b080a655e19c460a497d"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.627914 22220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:26.628573 22235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.628786 22220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:26.628873 22220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "5ace053942374a7193e2100f472011da"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "3b66ade72362e14f5b68ba6b03f55200"
server_key_iv: "d2362573eb55b080a655e19c460a497d"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.628935 22220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:26.657835 22220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:26.659940 22220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:26.660077 22220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:26.660303 22220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:26.660594 22220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:26.660627 22220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.660657 22220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:26.660676 22220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.665920 22220 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:33435
I20260928 17:50:26.665968 22348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:33435 every 8 connection(s)
I20260928 17:50:26.666256 22220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:50:26.668779 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 22220
I20260928 17:50:26.668888 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:50:26.669062 3960 external_mini_cluster.cc:1428] Setting key 114c87cd0948cb657142904129df782a
Sep 28 17:50:26 dist-test-slave-w3bm krb5kdc[21979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:26.676066 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.666954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:54551 (local address 127.3.222.62:35735)
0928 17:50:26.666997 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:26.666999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:26.667188 (+ 189us) server_negotiation.cc:407] Connection header received
0928 17:50:26.667716 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:26.667718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:26.667728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:26.667793 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:26.668893 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.669588 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.670923 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.671186 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.671276 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.673573 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:26.673575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:26.673575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:26.673583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:26.674819 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.675162 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.675163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.675163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.675247 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.675468 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.675469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.675469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.675576 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:26.675639 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:26.675930 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 17:50:26.676004 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:50:26.676493 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.666918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35735 (local address 127.3.222.2:54551)
0928 17:50:26.667072 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:26.667113 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:26.667603 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:26.667899 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:26.667903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:26.668459 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:26.668658 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.668663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.669677 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.669678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.670836 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.670839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.671282 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.671283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.671391 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.671912 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:26.671923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:26.673489 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:26.674887 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:26.674894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:26.674896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:26.675099 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:26.675301 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:26.675302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:26.675302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:26.675412 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:26.675639 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:26.675645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:26.675776 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 17:50:26.676152 (+ 376us) client_negotiation.cc:240] Negotiation successful
0928 17:50:26.676327 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
I20260928 17:50:26.676935 22349 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35735
I20260928 17:50:26.677018 22349 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:26.677181 22349 heartbeater.cc:511] Master 127.3.222.62:35735 requested a full tablet report, sending...
I20260928 17:50:26.677515 22024 ts_manager.cc:194] Registered new tserver with Master: 5ace053942374a7193e2100f472011da (127.3.222.2:33435)
I20260928 17:50:26.677901 22024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:54551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:50:26.692162 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35735
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:26.769060 22354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:26.769296 22354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:26.769322 22354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:26.770509 22354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:26.770555 22354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:26.770589 22354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:50:26.772039 22354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:26.772209 22354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:26.772372 22354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:26.777048 22359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:26.777055 22360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:26.777055 22362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:26.777246 22354 server_base.cc:1034] running on GCE node
I20260928 17:50:26.777487 22354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:26.777731 22354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:26.778903 22354 hybrid_clock.cc:648] HybridClock initialized: now 1790617826778888 us; error 30 us; skew 500 ppm
Sep 28 17:50:26 dist-test-slave-w3bm krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:26.781495 22354 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:50:26.781980 22354 webserver.cc:460] Webserver started at http://127.3.222.3:35895/ using document root <none> and password file <none>
I20260928 17:50:26.782204 22354 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:26.782267 22354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:26.782382 22354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:26.783496 22354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "8d840aa45c714a9ca0d551267c04f6f2"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "40861fa18168ac6e0f9af00de788f55a"
server_key_iv: "b41312f8a45cfcf5bd770bbb19722cf4"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.783825 22354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "8d840aa45c714a9ca0d551267c04f6f2"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "40861fa18168ac6e0f9af00de788f55a"
server_key_iv: "b41312f8a45cfcf5bd770bbb19722cf4"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.785064 22354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:26.785768 22369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.785953 22354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.786036 22354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "8d840aa45c714a9ca0d551267c04f6f2"
format_stamp: "Formatted at 2026-09-28 17:50:26 on dist-test-slave-w3bm"
server_key: "40861fa18168ac6e0f9af00de788f55a"
server_key_iv: "b41312f8a45cfcf5bd770bbb19722cf4"
server_key_version: "encryptionkey@0"
I20260928 17:50:26.786101 22354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:26.806454 22354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:26.808214 22354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:26.808347 22354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:26.808550 22354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:26.808879 22354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:26.808917 22354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.808946 22354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:26.808970 22354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:26.814172 22354 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:33755
I20260928 17:50:26.814239 22482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:33755 every 8 connection(s)
I20260928 17:50:26.814508 22354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:50:26.816278 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 22354
I20260928 17:50:26.816386 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:50:26.816574 3960 external_mini_cluster.cc:1428] Setting key 6aac358bab42864425b0da27cda2df70
Sep 28 17:50:26 dist-test-slave-w3bm krb5kdc[21979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:26.823429 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.815231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:59921 (local address 127.3.222.62:35735)
0928 17:50:26.815278 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:26.815280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:26.815459 (+ 179us) server_negotiation.cc:407] Connection header received
0928 17:50:26.815982 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:26.815983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:26.815996 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:26.816044 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:26.817252 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.817886 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.818857 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.819044 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.819097 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.821032 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:26.821033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:26.821033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:26.821040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:26.822215 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.822572 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.822573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.822573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.822622 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.822800 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.822800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.822801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.822903 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:26.822954 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:26.823284 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 17:50:26.823344 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:50:26.823578 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.815165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:35735 (local address 127.3.222.3:59921)
0928 17:50:26.815343 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:26.815381 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:26.815856 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:26.816132 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:26.816137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:26.816847 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:26.817135 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.817142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.817982 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.817983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.818773 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.818776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.819107 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.819108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.819204 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.819711 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:26.819719 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:26.820959 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:26.822306 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:26.822312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:26.822314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:26.822514 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:26.822668 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:26.822668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:26.822669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:26.822752 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:26.822986 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:26.822992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:26.823109 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 17:50:26.823249 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 17:50:26.823384 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260928 17:50:26.823987 22483 heartbeater.cc:347] Connected to a master server at 127.3.222.62:35735
I20260928 17:50:26.824062 22483 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:26.824213 22483 heartbeater.cc:511] Master 127.3.222.62:35735 requested a full tablet report, sending...
I20260928 17:50:26.824571 22024 ts_manager.cc:194] Registered new tserver with Master: 8d840aa45c714a9ca0d551267c04f6f2 (127.3.222.3:33755)
I20260928 17:50:26.824930 22024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:59921
I20260928 17:50:26.828243 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:50:26.835060 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.829439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.3.222.62:35735)
0928 17:50:26.829479 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:26.829481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:26.829601 (+ 120us) server_negotiation.cc:407] Connection header received
0928 17:50:26.829663 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:26.829664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:26.829675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:26.829743 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:26.830300 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.830762 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.831481 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.831697 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.831757 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.832859 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:26.832861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:26.832861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:26.832867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:26.834093 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.834412 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.834412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.834412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.834456 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:26.834671 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:26.834672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:26.834672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:26.834787 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:26.834856 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:26.834945 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 17:50:26.834990 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:50:26.836536 22024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:50:26.838056 22024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:50:26.844331 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.840247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:33435 (local address 127.0.0.1:51206)
0928 17:50:26.840696 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:26.840702 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:26.840735 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:26.840910 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:26.840911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:26.840913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:26.840987 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.840989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.842341 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.842342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.843422 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.843424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.844042 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.844043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.844119 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.844132 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:50:26.844199 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:50:26.844257 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":4,"thread_start_us":111,"threads_started":1}
I20260928 17:50:26.844357 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.840442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.3.222.2:33435)
0928 17:50:26.840627 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:26.840631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:26.840753 (+ 122us) server_negotiation.cc:407] Connection header received
0928 17:50:26.840793 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:26.840795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:26.840808 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:26.840878 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:26.841101 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.842245 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.843486 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.843969 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.844030 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.844090 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:26.844236 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 17:50:26.844286 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260928 17:50:26.844662 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.840430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.3.222.1:46237)
0928 17:50:26.840691 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:26.840694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:26.840702 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:50:26.840736 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:26.840739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:26.840754 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:26.840854 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:26.841326 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.842006 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.843441 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.844201 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.844255 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.844311 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:26.844538 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 17:50:26.844585 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":82,"threads_started":1}
I20260928 17:50:26.844780 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.840459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.3.222.3:33755)
0928 17:50:26.841039 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:26.841043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:26.841052 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:50:26.841090 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:26.841092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:26.841107 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:26.841213 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:26.841372 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.842416 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.843480 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.844168 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.844236 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.844293 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:26.844665 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 17:50:26.844707 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":76,"threads_started":1}
I20260928 17:50:26.844589 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.840243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:33755 (local address 127.0.0.1:57296)
0928 17:50:26.840420 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:26.840459 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:26.840521 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:26.841211 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:26.841212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:26.841214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:26.841250 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.841252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.842497 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.842498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.843385 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.843388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.844328 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.844329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.844394 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.844402 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 17:50:26.844447 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 17:50:26.844495 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":7,"thread_start_us":55,"threads_started":1}
I20260928 17:50:26.844650 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:26.840245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:46237 (local address 127.0.0.1:36210)
0928 17:50:26.840607 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:26.840615 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:26.840652 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:26.840907 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:26.840911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:26.840915 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:26.840988 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.840990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.842099 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.842100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.843364 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:26.843367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:26.844338 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:26.844339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:26.844394 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:26.844400 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:50:26.844447 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 17:50:26.844495 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":6,"spinlock_wait_cycles":8448,"thread_start_us":112,"threads_started":1}
I20260928 17:50:26.845510 22417 tablet_service.cc:1467] Processing CreateTablet for tablet e0b54caa07284d8fb0a835b6ffe79441 (DEFAULT_TABLE table=test-table [id=d9c8301e62134d7795838ed20964a576]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:26.845510 22283 tablet_service.cc:1467] Processing CreateTablet for tablet e0b54caa07284d8fb0a835b6ffe79441 (DEFAULT_TABLE table=test-table [id=d9c8301e62134d7795838ed20964a576]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:26.845851 22417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b54caa07284d8fb0a835b6ffe79441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:26.845851 22283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b54caa07284d8fb0a835b6ffe79441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:26.847738 22149 tablet_service.cc:1467] Processing CreateTablet for tablet e0b54caa07284d8fb0a835b6ffe79441 (DEFAULT_TABLE table=test-table [id=d9c8301e62134d7795838ed20964a576]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:26.848033 22149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b54caa07284d8fb0a835b6ffe79441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:26.849086 22501 tablet_bootstrap.cc:492] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: Bootstrap starting.
I20260928 17:50:26.849294 22500 tablet_bootstrap.cc:492] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: Bootstrap starting.
I20260928 17:50:26.849979 22500 tablet_bootstrap.cc:654] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:26.850535 22500 log.cc:824] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:26.850940 22503 tablet_bootstrap.cc:492] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0: Bootstrap starting.
I20260928 17:50:26.851320 22500 tablet_bootstrap.cc:492] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: No bootstrap required, opened a new log
I20260928 17:50:26.851389 22500 ts_tablet_manager.cc:1397] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:50:26.851397 22501 tablet_bootstrap.cc:654] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:26.851708 22503 tablet_bootstrap.cc:654] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:26.851848 22501 log.cc:824] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:26.852206 22503 log.cc:824] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:26.852475 22501 tablet_bootstrap.cc:492] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: No bootstrap required, opened a new log
I20260928 17:50:26.852545 22501 ts_tablet_manager.cc:1397] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 17:50:26.852905 22503 tablet_bootstrap.cc:492] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0: No bootstrap required, opened a new log
I20260928 17:50:26.852972 22503 ts_tablet_manager.cc:1397] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:26.853340 22500 raft_consensus.cc:348] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } }
I20260928 17:50:26.853513 22500 raft_consensus.cc:374] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:26.853544 22500 raft_consensus.cc:729] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ace053942374a7193e2100f472011da, State: Initialized, Role: FOLLOWER
I20260928 17:50:26.853677 22500 consensus_queue.cc:260] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } }
I20260928 17:50:26.853971 22500 ts_tablet_manager.cc:1428] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:50:26.854084 22349 heartbeater.cc:503] Master 127.3.222.62:35735 was elected leader, sending a full tablet report...
I20260928 17:50:26.854055 22501 raft_consensus.cc:348] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } }
I20260928 17:50:26.854215 22501 raft_consensus.cc:374] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:26.854255 22501 raft_consensus.cc:729] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d840aa45c714a9ca0d551267c04f6f2, State: Initialized, Role: FOLLOWER
I20260928 17:50:26.854404 22501 consensus_queue.cc:260] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } }
I20260928 17:50:26.854661 22501 ts_tablet_manager.cc:1428] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:50:26.854682 22483 heartbeater.cc:503] Master 127.3.222.62:35735 was elected leader, sending a full tablet report...
I20260928 17:50:26.855064 22503 raft_consensus.cc:348] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } }
I20260928 17:50:26.855242 22503 raft_consensus.cc:374] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:26.855279 22503 raft_consensus.cc:729] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2730d5fa671a4d1c951f1ea9d27c75f0, State: Initialized, Role: FOLLOWER
I20260928 17:50:26.855387 22503 consensus_queue.cc:260] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } }
I20260928 17:50:26.855634 22503 ts_tablet_manager.cc:1428] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:50:26.855679 22215 heartbeater.cc:503] Master 127.3.222.62:35735 was elected leader, sending a full tablet report...
W20260928 17:50:26.916836 22350 tablet.cc:2369] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:50:27.000252 22216 tablet.cc:2369] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:50:27.028108 22507 raft_consensus.cc:482] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:50:27.028232 22507 raft_consensus.cc:504] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } }
I20260928 17:50:27.028591 22507 leader_election.cc:288] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ace053942374a7193e2100f472011da (127.3.222.2:33435), 2730d5fa671a4d1c951f1ea9d27c75f0 (127.3.222.1:46237)
I20260928 17:50:27.031283 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.028820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:40639 (local address 127.3.222.1:46237)
0928 17:50:27.028861 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.028863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.028873 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:50:27.028944 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.028944 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.028955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.029024 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:27.029152 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.029657 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.030568 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.030947 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.031005 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.031015 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.031154 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.031193 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:50:27.031291 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.028768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:46237 (local address 127.3.222.3:40639)
0928 17:50:27.028834 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.028845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.028887 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.029029 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.029030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.029031 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:27.029098 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.029100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.029731 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.029732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.030487 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.030489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.031011 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.031011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.031094 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.031097 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.031147 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.031187 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 17:50:27.031333 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.028851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:48325 (local address 127.3.222.2:33435)
0928 17:50:27.028899 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.028901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.029027 (+ 126us) server_negotiation.cc:407] Connection header received
0928 17:50:27.029077 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.029078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.029090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.029136 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:27.029266 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.029765 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.030623 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.031007 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.031086 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.031095 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.031236 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.031276 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:50:27.031363 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.028778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:33435 (local address 127.3.222.3:48325)
0928 17:50:27.028956 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.028964 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.029025 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.029142 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.029143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.029144 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:27.029211 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.029212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.029834 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.029835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.030547 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.030549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.031093 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.031093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.031155 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.031158 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.031225 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.031255 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260928 17:50:27.031646 22169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b54caa07284d8fb0a835b6ffe79441" candidate_uuid: "8d840aa45c714a9ca0d551267c04f6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" is_pre_election: true
I20260928 17:50:27.031834 22169 raft_consensus.cc:2454] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d840aa45c714a9ca0d551267c04f6f2 in term 0.
I20260928 17:50:27.031831 22303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b54caa07284d8fb0a835b6ffe79441" candidate_uuid: "8d840aa45c714a9ca0d551267c04f6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ace053942374a7193e2100f472011da" is_pre_election: true
I20260928 17:50:27.031991 22303 raft_consensus.cc:2454] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d840aa45c714a9ca0d551267c04f6f2 in term 0.
I20260928 17:50:27.032058 22373 leader_election.cc:302] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2730d5fa671a4d1c951f1ea9d27c75f0, 8d840aa45c714a9ca0d551267c04f6f2; no voters:
I20260928 17:50:27.032186 22507 raft_consensus.cc:2781] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:50:27.032232 22507 raft_consensus.cc:482] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:50:27.032256 22507 raft_consensus.cc:3037] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:27.032928 22507 raft_consensus.cc:504] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } }
I20260928 17:50:27.033041 22507 leader_election.cc:288] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ace053942374a7193e2100f472011da (127.3.222.2:33435), 2730d5fa671a4d1c951f1ea9d27c75f0 (127.3.222.1:46237)
I20260928 17:50:27.033171 22303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b54caa07284d8fb0a835b6ffe79441" candidate_uuid: "8d840aa45c714a9ca0d551267c04f6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ace053942374a7193e2100f472011da"
I20260928 17:50:27.033241 22303 raft_consensus.cc:3037] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:27.033298 22169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b54caa07284d8fb0a835b6ffe79441" candidate_uuid: "8d840aa45c714a9ca0d551267c04f6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0"
I20260928 17:50:27.033391 22169 raft_consensus.cc:3037] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:27.033895 22303 raft_consensus.cc:2454] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d840aa45c714a9ca0d551267c04f6f2 in term 1.
I20260928 17:50:27.034053 22370 leader_election.cc:302] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ace053942374a7193e2100f472011da, 8d840aa45c714a9ca0d551267c04f6f2; no voters:
I20260928 17:50:27.034142 22507 raft_consensus.cc:2781] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:27.034185 22169 raft_consensus.cc:2454] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d840aa45c714a9ca0d551267c04f6f2 in term 1.
I20260928 17:50:27.034289 22507 raft_consensus.cc:686] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 1 LEADER]: Becoming Leader. State: Replica: 8d840aa45c714a9ca0d551267c04f6f2, State: Running, Role: LEADER
I20260928 17:50:27.034385 22507 consensus_queue.cc:237] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } }
I20260928 17:50:27.035164 22023 catalog_manager.cc:5495] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d840aa45c714a9ca0d551267c04f6f2 (127.3.222.3). New cstate: current_term: 1 leader_uuid: "8d840aa45c714a9ca0d551267c04f6f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d840aa45c714a9ca0d551267c04f6f2" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33755 } health_report { overall_health: HEALTHY } } }
W20260928 17:50:27.065060 22484 tablet.cc:2369] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:50:27.072767 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.070459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.3.222.3:33755)
0928 17:50:27.070512 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.070514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.070533 (+ 19us) server_negotiation.cc:407] Connection header received
0928 17:50:27.070571 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.070572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.070594 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.070651 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:27.070783 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.071285 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.072003 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.072204 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.072257 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.072421 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:27.072519 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:27.072608 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.072660 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.072668 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:50:27.074468 22169 raft_consensus.cc:1261] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [term 1 FOLLOWER]: Refusing update from remote peer 8d840aa45c714a9ca0d551267c04f6f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:27.074468 22303 raft_consensus.cc:1261] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [term 1 FOLLOWER]: Refusing update from remote peer 8d840aa45c714a9ca0d551267c04f6f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:27.074819 22507 consensus_queue.cc:1035] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2730d5fa671a4d1c951f1ea9d27c75f0" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:27.074959 22510 consensus_queue.cc:1035] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ace053942374a7193e2100f472011da" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:27.077909 22515 mvcc.cc:205] Tried to move back new op lower bound from 7334370619694395392 to 7334370619533406208. Current Snapshot: MvccSnapshot[applied={T|T < 7334370619694395392 or (T in {7334370619694395392})}]
I20260928 17:50:27.079988 22024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:27.080089 22024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:27.080808 22024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab01ff388cdf49a39c1b332fda5bdd2f: Sending DeleteTablet for 3 replicas of tablet e0b54caa07284d8fb0a835b6ffe79441
I20260928 17:50:27.081184 22149 tablet_service.cc:1514] Processing DeleteTablet for tablet e0b54caa07284d8fb0a835b6ffe79441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:27 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:36210
I20260928 17:50:27.081213 22283 tablet_service.cc:1514] Processing DeleteTablet for tablet e0b54caa07284d8fb0a835b6ffe79441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:27 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:51206
I20260928 17:50:27.081436 22523 tablet_replica.cc:330] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: stopping tablet replica
I20260928 17:50:27.081578 22523 raft_consensus.cc:2229] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:27.081554 22417 tablet_service.cc:1514] Processing DeleteTablet for tablet e0b54caa07284d8fb0a835b6ffe79441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:27 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:57296
I20260928 17:50:27.081753 22523 raft_consensus.cc:2258] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:27.081781 22522 tablet_replica.cc:330] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0: stopping tablet replica
I20260928 17:50:27.081885 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 22086
I20260928 17:50:27.081912 22522 raft_consensus.cc:2229] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:27.082010 22522 raft_consensus.cc:2258] T e0b54caa07284d8fb0a835b6ffe79441 P 2730d5fa671a4d1c951f1ea9d27c75f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:27.082986 22523 ts_tablet_manager.cc:1905] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:50:27.084565 22524 tablet_replica.cc:330] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: stopping tablet replica
I20260928 17:50:27.084653 22524 raft_consensus.cc:2229] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:50:27.084848 22523 ts_tablet_manager.cc:1918] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:27.084915 22523 log.cc:1196] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/e0b54caa07284d8fb0a835b6ffe79441
I20260928 17:50:27.085144 22523 ts_tablet_manager.cc:1939] T e0b54caa07284d8fb0a835b6ffe79441 P 5ace053942374a7193e2100f472011da: Deleting consensus metadata
I20260928 17:50:27.085261 22524 raft_consensus.cc:2258] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:27.085608 22008 catalog_manager.cc:4841] TS 5ace053942374a7193e2100f472011da (127.3.222.2:33435): tablet e0b54caa07284d8fb0a835b6ffe79441 (table test-table [id=d9c8301e62134d7795838ed20964a576]) successfully deleted
I20260928 17:50:27.085958 22524 ts_tablet_manager.cc:1905] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:50:27.087440 22524 ts_tablet_manager.cc:1918] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:27.087534 22524 log.cc:1196] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/e0b54caa07284d8fb0a835b6ffe79441
I20260928 17:50:27.087781 22524 ts_tablet_manager.cc:1939] T e0b54caa07284d8fb0a835b6ffe79441 P 8d840aa45c714a9ca0d551267c04f6f2: Deleting consensus metadata
I20260928 17:50:27.088297 22009 catalog_manager.cc:4841] TS 8d840aa45c714a9ca0d551267c04f6f2 (127.3.222.3:33755): tablet e0b54caa07284d8fb0a835b6ffe79441 (table test-table [id=d9c8301e62134d7795838ed20964a576]) successfully deleted
W20260928 17:50:27.088683 22011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:46237 (error 108)
I20260928 17:50:27.089172 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.089038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:46237 (local address 127.0.0.1:36224)
0928 17:50:27.089103 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.089139 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 17:50:27.089182 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 22220
I20260928 17:50:27.093173 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 22354
W20260928 17:50:27.093209 22011 catalog_manager.cc:4584] TS 2730d5fa671a4d1c951f1ea9d27c75f0 (127.3.222.1:46237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0b54caa07284d8fb0a835b6ffe79441: Network error: Client connection negotiation failed: client connection to 127.3.222.1:46237: connect: Connection refused (error 111)
I20260928 17:50:27.097045 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 21992
2026-09-28T17:50:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2010 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T17:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:27Z Disabled control of system clock
I20260928 17:50:27.121472 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41789
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:27.198143 22529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:27.198357 22529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:27.198385 22529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:27.199532 22529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:27.199570 22529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:27.199587 22529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:27.199604 22529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:27.199620 22529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:27.200930 22529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:27.201161 22529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:27.201424 22529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:50:27.205960 22529 server_base.cc:1034] running on GCE node
W20260928 17:50:27.205919 22535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:27.206025 22537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:27.205948 22534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:27.206408 22529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:27.206650 22529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:27.207780 22529 hybrid_clock.cc:648] HybridClock initialized: now 1790617827207766 us; error 30 us; skew 500 ppm
I20260928 17:50:27.208909 22529 webserver.cc:460] Webserver started at http://127.3.222.62:40079/ using document root <none> and password file <none>
I20260928 17:50:27.209107 22529 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:27.209154 22529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:27.209271 22529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:27.210238 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "d6d13db7bc7a47249befe4ade7ef293c"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "06ae7b082c0ca9a28778d083a52de688"
server_key_iv: "0e31d3271004866824171971ae46b606"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.210546 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "d6d13db7bc7a47249befe4ade7ef293c"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "06ae7b082c0ca9a28778d083a52de688"
server_key_iv: "0e31d3271004866824171971ae46b606"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.211818 22529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:27.212514 22543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.212687 22529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:27.212826 22529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "d6d13db7bc7a47249befe4ade7ef293c"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "06ae7b082c0ca9a28778d083a52de688"
server_key_iv: "0e31d3271004866824171971ae46b606"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.212889 22529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:27.239853 22529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:27.240157 22529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:27.240280 22529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:27.244035 22529 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:41789
I20260928 17:50:27.244055 22595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:41789 every 8 connection(s)
I20260928 17:50:27.244388 22529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:27.245163 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 22529
I20260928 17:50:27.245244 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:27.245298 22596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:27.245419 3960 external_mini_cluster.cc:1428] Setting key 2c84512206268388ad52faa98f07cca2
I20260928 17:50:27.247951 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c: Bootstrap starting.
I20260928 17:50:27.248493 22596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:27.248731 22596 log.cc:824] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:27.249311 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c: No bootstrap required, opened a new log
I20260928 17:50:27.250399 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.245908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.3.222.62:41789)
0928 17:50:27.246105 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.246112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.246136 (+ 24us) server_negotiation.cc:407] Connection header received
0928 17:50:27.246594 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.246600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.246664 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.246788 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:27.246985 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.247879 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.248683 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.249121 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.249247 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.249847 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:27.249851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:27.249851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:27.250002 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:27.250032 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:27.250042 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:27.250094 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.250166 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.250248 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260928 17:50:27.251189 22596 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d13db7bc7a47249befe4ade7ef293c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41789 } }
I20260928 17:50:27.251358 22596 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:27.251394 22596 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d13db7bc7a47249befe4ade7ef293c, State: Initialized, Role: FOLLOWER
I20260928 17:50:27.251505 22596 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d13db7bc7a47249befe4ade7ef293c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41789 } }
I20260928 17:50:27.251566 22596 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:27.251595 22596 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:27.251641 22596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:27.252127 22596 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d13db7bc7a47249befe4ade7ef293c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41789 } }
I20260928 17:50:27.252218 22596 leader_election.cc:302] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6d13db7bc7a47249befe4ade7ef293c; no voters:
I20260928 17:50:27.252377 22596 leader_election.cc:288] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:27.252418 22601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:27.252522 22601 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [term 1 LEADER]: Becoming Leader. State: Replica: d6d13db7bc7a47249befe4ade7ef293c, State: Running, Role: LEADER
I20260928 17:50:27.252628 22596 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:27.252655 22601 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d13db7bc7a47249befe4ade7ef293c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41789 } }
I20260928 17:50:27.253001 22602 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6d13db7bc7a47249befe4ade7ef293c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d13db7bc7a47249befe4ade7ef293c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41789 } } }
I20260928 17:50:27.253008 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d13db7bc7a47249befe4ade7ef293c. Latest consensus state: current_term: 1 leader_uuid: "d6d13db7bc7a47249befe4ade7ef293c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d13db7bc7a47249befe4ade7ef293c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41789 } } }
I20260928 17:50:27.253188 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:27.253496 22602 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:27.253563 22609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:27.254032 22609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:27.255350 22609 catalog_manager.cc:1347] Generated new cluster ID: d003dcb49a684886af46c588aa895cdb
I20260928 17:50:27.255393 22609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:27.261204 22609 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:27.261660 22609 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:27.268849 22609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c: Generated new TSK 0
I20260928 17:50:27.269023 22609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:27.272377 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41789
--builtin_ntp_servers=127.3.222.20:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:27.352170 22620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:27.352403 22620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:27.352429 22620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:27.353780 22620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:27.353827 22620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:27.353852 22620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:50:27.355345 22620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:27.355571 22620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:27.355867 22620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:27.360675 22628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:27.360683 22625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:27.360961 22620 server_base.cc:1034] running on GCE node
W20260928 17:50:27.361071 22626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:27.361300 22620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:27.361529 22620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:27.362671 22620 hybrid_clock.cc:648] HybridClock initialized: now 1790617827362658 us; error 28 us; skew 500 ppm
I20260928 17:50:27.363839 22620 webserver.cc:460] Webserver started at http://127.3.222.1:41491/ using document root <none> and password file <none>
I20260928 17:50:27.364053 22620 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:27.364100 22620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:27.364226 22620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:27.365284 22620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "5bc6d073b86141c8a5d3516577f787f2"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "c37aec37ab1cf2ee2b4923e3dad1dd91"
server_key_iv: "f06fe12d06a6f3dc1b48c1f3edd67345"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.365613 22620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "5bc6d073b86141c8a5d3516577f787f2"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "c37aec37ab1cf2ee2b4923e3dad1dd91"
server_key_iv: "f06fe12d06a6f3dc1b48c1f3edd67345"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.366843 22620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:50:27.367556 22634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.367707 22620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.367787 22620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "5bc6d073b86141c8a5d3516577f787f2"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "c37aec37ab1cf2ee2b4923e3dad1dd91"
server_key_iv: "f06fe12d06a6f3dc1b48c1f3edd67345"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.367856 22620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:27.378748 22620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:27.379021 22620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:27.379137 22620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:27.379361 22620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:27.379668 22620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:27.379701 22620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.379721 22620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:27.379734 22620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.385198 22620 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:37061
I20260928 17:50:27.385250 22747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:37061 every 8 connection(s)
I20260928 17:50:27.385538 22620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:50:27.387928 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 22620
I20260928 17:50:27.388039 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:50:27.388211 3960 external_mini_cluster.cc:1428] Setting key e950c61d8136d8c4016309c9f0fbf7bb
I20260928 17:50:27.389060 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41789
--builtin_ntp_servers=127.3.222.20:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 17:50:27.390813 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.386179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:35371 (local address 127.3.222.62:41789)
0928 17:50:27.386234 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.386236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.386468 (+ 232us) server_negotiation.cc:407] Connection header received
0928 17:50:27.386934 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.386935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.386949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.386988 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:27.387637 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.388359 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.389502 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.389763 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.389831 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.390599 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:27.390600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:27.390601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:27.390605 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:27.390619 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:27.390622 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:27.390660 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.390726 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.390750 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:50:27.391036 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.386177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41789 (local address 127.3.222.1:35371)
0928 17:50:27.386352 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.386384 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.386849 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.387051 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.387054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.387062 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:27.387577 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.387582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.388445 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.388446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.389427 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.389430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.389831 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.389831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.389934 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.390453 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:50:27.390454 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:27.390493 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:50:27.390521 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:27.390662 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:27.390665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:27.390668 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.390727 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.390824 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260928 17:50:27.391544 22748 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41789
I20260928 17:50:27.391639 22748 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:27.391811 22748 heartbeater.cc:511] Master 127.3.222.62:41789 requested a full tablet report, sending...
I20260928 17:50:27.392320 22560 ts_manager.cc:194] Registered new tserver with Master: 5bc6d073b86141c8a5d3516577f787f2 (127.3.222.1:37061)
I20260928 17:50:27.393193 22560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.1:35371
W20260928 17:50:27.467748 22751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:27.467975 22751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:27.467998 22751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:27.469099 22751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:27.469135 22751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:27.469158 22751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:50:27.470465 22751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:27.470652 22751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:27.470947 22751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:27.475829 22756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:27.475948 22751 server_base.cc:1034] running on GCE node
W20260928 17:50:27.476105 22757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:27.475951 22759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:27.476406 22751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:27.476665 22751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:27.477800 22751 hybrid_clock.cc:648] HybridClock initialized: now 1790617827477784 us; error 33 us; skew 500 ppm
I20260928 17:50:27.479010 22751 webserver.cc:460] Webserver started at http://127.3.222.2:38743/ using document root <none> and password file <none>
I20260928 17:50:27.479262 22751 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:27.479311 22751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:27.479455 22751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:27.480489 22751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "38930024892b4ec7b142e1c08652a7d1"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "b7afd8e9a00c53e92a78c250c1023c04"
server_key_iv: "0d350e68a3f6c445b4bb6f619a70178c"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.480803 22751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "38930024892b4ec7b142e1c08652a7d1"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "b7afd8e9a00c53e92a78c250c1023c04"
server_key_iv: "0d350e68a3f6c445b4bb6f619a70178c"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.482010 22751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:50:27.482671 22765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.482802 22751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:50:27.482877 22751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "38930024892b4ec7b142e1c08652a7d1"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "b7afd8e9a00c53e92a78c250c1023c04"
server_key_iv: "0d350e68a3f6c445b4bb6f619a70178c"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.482944 22751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:27.499194 22751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:27.499519 22751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:27.499643 22751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:27.499861 22751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:27.500172 22751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:27.500206 22751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.500228 22751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:27.500250 22751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.505510 22751 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:35285
I20260928 17:50:27.505576 22878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:35285 every 8 connection(s)
I20260928 17:50:27.505879 22751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:50:27.510749 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.506541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:37173 (local address 127.3.222.62:41789)
0928 17:50:27.506593 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.506595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.506802 (+ 207us) server_negotiation.cc:407] Connection header received
0928 17:50:27.507330 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.507332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.507343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.507402 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:27.508092 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.508593 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.509534 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.509716 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.509769 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.510504 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:27.510505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:27.510506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:27.510511 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:27.510524 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:27.510528 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:27.510569 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.510644 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.510669 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:50:27.510958 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.506524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41789 (local address 127.3.222.2:37173)
0928 17:50:27.506688 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.506721 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.507240 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.507470 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.507474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.507485 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:27.508004 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.508008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.508671 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.508672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.509452 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.509453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.509780 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.509781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.509911 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.510378 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:50:27.510378 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:27.510408 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:50:27.510429 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:27.510574 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:27.510578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:27.510581 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.510640 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.510738 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260928 17:50:27.511401 22879 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41789
I20260928 17:50:27.511490 22879 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:27.511659 22879 heartbeater.cc:511] Master 127.3.222.62:41789 requested a full tablet report, sending...
I20260928 17:50:27.511981 22560 ts_manager.cc:194] Registered new tserver with Master: 38930024892b4ec7b142e1c08652a7d1 (127.3.222.2:35285)
I20260928 17:50:27.512301 22560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.2:37173
I20260928 17:50:27.514084 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 22751
I20260928 17:50:27.514178 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:50:27.514365 3960 external_mini_cluster.cc:1428] Setting key 9d85f2c38a2679c30052e87aeb28162e
I20260928 17:50:27.515175 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41789
--builtin_ntp_servers=127.3.222.20:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:27.592325 22882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:27.592538 22882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:27.592558 22882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:27.593626 22882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:27.593657 22882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:27.593678 22882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:50:27.594945 22882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:41789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:27.595121 22882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:27.595368 22882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:27.599956 22888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:27.599979 22890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:27.599995 22887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:27.600167 22882 server_base.cc:1034] running on GCE node
I20260928 17:50:27.600381 22882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:27.600577 22882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:27.601717 22882 hybrid_clock.cc:648] HybridClock initialized: now 1790617827601701 us; error 30 us; skew 500 ppm
I20260928 17:50:27.609237 22882 webserver.cc:460] Webserver started at http://127.3.222.3:36847/ using document root <none> and password file <none>
I20260928 17:50:27.609469 22882 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:27.609515 22882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:27.609640 22882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:27.610694 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "c034cde3ce64475cac13699dd7eeaf25"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "6db58a12b0c4de94a09efacb21051de3"
server_key_iv: "93020f9f2990846a21446949206270fa"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.611008 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "c034cde3ce64475cac13699dd7eeaf25"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "6db58a12b0c4de94a09efacb21051de3"
server_key_iv: "93020f9f2990846a21446949206270fa"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.612270 22882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:27.612946 22896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.613086 22882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:27.613173 22882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "c034cde3ce64475cac13699dd7eeaf25"
format_stamp: "Formatted at 2026-09-28 17:50:27 on dist-test-slave-w3bm"
server_key: "6db58a12b0c4de94a09efacb21051de3"
server_key_iv: "93020f9f2990846a21446949206270fa"
server_key_version: "encryptionkey@0"
I20260928 17:50:27.613262 22882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:27.645428 22882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:27.645733 22882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:27.645860 22882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:27.646086 22882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:27.646404 22882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:27.646437 22882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.646466 22882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:27.646483 22882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:27.651760 22882 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:34071
I20260928 17:50:27.651798 23009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:34071 every 8 connection(s)
I20260928 17:50:27.652153 22882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:50:27.657013 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.652794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:59205 (local address 127.3.222.62:41789)
0928 17:50:27.652838 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.652840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.653089 (+ 249us) server_negotiation.cc:407] Connection header received
0928 17:50:27.653604 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.653606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.653625 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.653682 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:27.654344 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.654875 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.655863 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.656053 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.656110 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.656780 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:27.656781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:27.656781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:27.656786 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:27.656799 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:27.656804 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:27.656842 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.656929 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.656954 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:50:27.657152 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.652777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41789 (local address 127.3.222.3:59205)
0928 17:50:27.652947 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.652980 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.653486 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.653762 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.653773 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.653782 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:27.654261 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.654265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.654958 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.654958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.655743 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.655745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.656117 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.656117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.656222 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.656665 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:50:27.656665 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:27.656692 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:50:27.656712 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:27.656843 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:27.656846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:27.656848 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.656896 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.656972 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260928 17:50:27.657640 23010 heartbeater.cc:347] Connected to a master server at 127.3.222.62:41789
I20260928 17:50:27.657724 23010 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:27.658011 23010 heartbeater.cc:511] Master 127.3.222.62:41789 requested a full tablet report, sending...
I20260928 17:50:27.658397 22560 ts_manager.cc:194] Registered new tserver with Master: c034cde3ce64475cac13699dd7eeaf25 (127.3.222.3:34071)
I20260928 17:50:27.658792 22560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.3:59205
I20260928 17:50:27.659770 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 22882
I20260928 17:50:27.659847 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:50:27.659992 3960 external_mini_cluster.cc:1428] Setting key 479fa0389aeef4be8ab4d0e10b2f37c9
I20260928 17:50:27.660722 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:50:27.664355 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.661910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.3.222.62:41789)
0928 17:50:27.661950 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.661952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.662028 (+ 76us) server_negotiation.cc:407] Connection header received
0928 17:50:27.662067 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.662068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.662075 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.662116 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:27.662222 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.662658 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.663425 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.663604 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.663648 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.664150 (+ 502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:27.664151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:27.664151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:27.664155 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:27.664176 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:27.664180 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:27.664244 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.664294 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.664303 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:50:27.665719 22560 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:50:27.667164 22560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:50:27.673141 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.669148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:34071 (local address 127.0.0.1:52546)
0928 17:50:27.669314 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.669354 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.669419 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.669793 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.669796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.669799 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:27.669859 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.669861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.671164 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.671165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.672080 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.672083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.672860 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.672861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.672936 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.672949 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.673017 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.673068 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":50,"thread_start_us":86,"threads_started":1}
I20260928 17:50:27.673151 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.669227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.3.222.3:34071)
0928 17:50:27.669601 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.669604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.669612 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:50:27.669642 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.669643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.669654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.669712 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:27.669919 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.671067 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.672159 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.672781 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.672849 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.672933 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.673041 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.673076 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":87,"threads_started":1}
I20260928 17:50:27.673398 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.669148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37061 (local address 127.0.0.1:52316)
0928 17:50:27.669434 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.669443 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.669490 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.669857 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.669858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.669859 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:27.669887 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.669888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.670997 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.670998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.672413 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.672415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.673170 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.673171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.673228 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.673231 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.673298 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.673335 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":5,"thread_start_us":46,"threads_started":1}
I20260928 17:50:27.673436 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.669410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.3.222.1:37061)
0928 17:50:27.669646 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.669649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.669656 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:50:27.669695 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.669697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.669711 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.669786 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:27.669956 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.670928 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.672471 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.673003 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.673058 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.673138 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.673319 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.673359 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260928 17:50:27.673472 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.669322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:35285 (local address 127.0.0.1:49198)
0928 17:50:27.669779 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.669789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.669834 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.669968 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.669968 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.669970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:27.669997 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.669998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.671190 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.671191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.672331 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.672334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.673255 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.673257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.673325 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.673329 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.673375 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.673411 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260928 17:50:27.673637 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.669420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.3.222.2:35285)
0928 17:50:27.669581 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.669586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.669843 (+ 257us) server_negotiation.cc:407] Connection header received
0928 17:50:27.669885 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.669887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.669901 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.669972 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:27.670052 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.671125 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.672413 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.673082 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.673139 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.673221 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.673530 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.673567 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260928 17:50:27.674153 22682 tablet_service.cc:1467] Processing CreateTablet for tablet e1f7662537924307bd651dbef193fd08 (DEFAULT_TABLE table=test-table [id=b1b179fbb57f4bfbbe29f6a08b8a3f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:27.674152 22944 tablet_service.cc:1467] Processing CreateTablet for tablet e1f7662537924307bd651dbef193fd08 (DEFAULT_TABLE table=test-table [id=b1b179fbb57f4bfbbe29f6a08b8a3f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:27.674510 22682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f7662537924307bd651dbef193fd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:27.674510 22944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f7662537924307bd651dbef193fd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:27.674463 22813 tablet_service.cc:1467] Processing CreateTablet for tablet e1f7662537924307bd651dbef193fd08 (DEFAULT_TABLE table=test-table [id=b1b179fbb57f4bfbbe29f6a08b8a3f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:27.674734 22813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f7662537924307bd651dbef193fd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:27.677068 23024 tablet_bootstrap.cc:492] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1: Bootstrap starting.
I20260928 17:50:27.677692 23024 tablet_bootstrap.cc:654] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:27.677793 23026 tablet_bootstrap.cc:492] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: Bootstrap starting.
I20260928 17:50:27.678031 23024 log.cc:824] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:27.678364 23026 tablet_bootstrap.cc:654] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:27.678632 23026 log.cc:824] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:27.678743 23024 tablet_bootstrap.cc:492] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1: No bootstrap required, opened a new log
I20260928 17:50:27.678807 23024 ts_tablet_manager.cc:1397] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:27.678956 23025 tablet_bootstrap.cc:492] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2: Bootstrap starting.
I20260928 17:50:27.679272 23026 tablet_bootstrap.cc:492] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: No bootstrap required, opened a new log
I20260928 17:50:27.679337 23026 ts_tablet_manager.cc:1397] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:27.679659 23025 tablet_bootstrap.cc:654] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:27.679991 23025 log.cc:824] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:27.680766 23025 tablet_bootstrap.cc:492] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2: No bootstrap required, opened a new log
I20260928 17:50:27.680867 23025 ts_tablet_manager.cc:1397] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:50:27.681172 23024 raft_consensus.cc:348] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } }
I20260928 17:50:27.681363 23024 raft_consensus.cc:374] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:27.681399 23024 raft_consensus.cc:729] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38930024892b4ec7b142e1c08652a7d1, State: Initialized, Role: FOLLOWER
I20260928 17:50:27.681339 23026 raft_consensus.cc:348] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } }
I20260928 17:50:27.681511 23026 raft_consensus.cc:374] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:27.681557 23026 raft_consensus.cc:729] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c034cde3ce64475cac13699dd7eeaf25, State: Initialized, Role: FOLLOWER
I20260928 17:50:27.681563 23024 consensus_queue.cc:260] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } }
I20260928 17:50:27.681686 23026 consensus_queue.cc:260] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } }
I20260928 17:50:27.681828 22879 heartbeater.cc:503] Master 127.3.222.62:41789 was elected leader, sending a full tablet report...
I20260928 17:50:27.681965 23026 ts_tablet_manager.cc:1428] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 17:50:27.682117 23010 heartbeater.cc:503] Master 127.3.222.62:41789 was elected leader, sending a full tablet report...
I20260928 17:50:27.682456 23024 ts_tablet_manager.cc:1428] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 17:50:27.683126 23025 raft_consensus.cc:348] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } }
I20260928 17:50:27.683271 23025 raft_consensus.cc:374] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:27.683311 23025 raft_consensus.cc:729] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc6d073b86141c8a5d3516577f787f2, State: Initialized, Role: FOLLOWER
I20260928 17:50:27.683413 23025 consensus_queue.cc:260] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } }
I20260928 17:50:27.683602 23025 ts_tablet_manager.cc:1428] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:50:27.683668 22748 heartbeater.cc:503] Master 127.3.222.62:41789 was elected leader, sending a full tablet report...
I20260928 17:50:27.711040 23030 raft_consensus.cc:482] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:50:27.711153 23030 raft_consensus.cc:504] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } }
I20260928 17:50:27.711512 23030 leader_election.cc:288] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c034cde3ce64475cac13699dd7eeaf25 (127.3.222.3:34071), 5bc6d073b86141c8a5d3516577f787f2 (127.3.222.1:37061)
I20260928 17:50:27.715420 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.711744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:60323 (local address 127.3.222.1:37061)
0928 17:50:27.711817 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.711818 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.711987 (+ 169us) server_negotiation.cc:407] Connection header received
0928 17:50:27.712124 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.712125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.712138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.712183 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:27.712310 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.713035 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.714494 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.715047 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.715119 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.715128 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.715296 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.715336 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:50:27.715420 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.711733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:33611 (local address 127.3.222.3:34071)
0928 17:50:27.711773 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.711775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.711802 (+ 27us) server_negotiation.cc:407] Connection header received
0928 17:50:27.712017 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.712019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.712029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.712089 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:27.712297 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.713069 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.714495 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.715048 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.715119 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.715128 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.715318 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.715361 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:50:27.715417 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.711666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:34071 (local address 127.3.222.2:33611)
0928 17:50:27.711717 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.711729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.711816 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.712143 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.712144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.712146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:27.712215 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.712218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.713147 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.713148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.714397 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.714400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.715121 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.715122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.715201 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.715205 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.715311 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.715348 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":63}
I20260928 17:50:27.715425 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.711666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37061 (local address 127.3.222.2:60323)
0928 17:50:27.711840 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.711848 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:27.712038 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:27.712194 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:27.712194 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:27.712195 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:27.712251 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.712252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.713141 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.713143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.714409 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:27.714410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.715121 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.715122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:27.715201 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.715227 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 17:50:27.715287 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 17:50:27.715331 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":20,"thread_start_us":102,"threads_started":1}
I20260928 17:50:27.715816 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f7662537924307bd651dbef193fd08" candidate_uuid: "38930024892b4ec7b142e1c08652a7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c034cde3ce64475cac13699dd7eeaf25" is_pre_election: true
I20260928 17:50:27.715981 22964 raft_consensus.cc:2454] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38930024892b4ec7b142e1c08652a7d1 in term 0.
I20260928 17:50:27.716058 22702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f7662537924307bd651dbef193fd08" candidate_uuid: "38930024892b4ec7b142e1c08652a7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc6d073b86141c8a5d3516577f787f2" is_pre_election: true
I20260928 17:50:27.716231 22702 raft_consensus.cc:2454] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38930024892b4ec7b142e1c08652a7d1 in term 0.
I20260928 17:50:27.716238 22768 leader_election.cc:302] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38930024892b4ec7b142e1c08652a7d1, c034cde3ce64475cac13699dd7eeaf25; no voters:
I20260928 17:50:27.716390 23030 raft_consensus.cc:2781] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:50:27.716439 23030 raft_consensus.cc:482] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:50:27.716459 23030 raft_consensus.cc:3037] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:27.717216 23030 raft_consensus.cc:504] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } }
I20260928 17:50:27.717430 23030 leader_election.cc:288] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [CANDIDATE]: Term 1 election: Requested vote from peers c034cde3ce64475cac13699dd7eeaf25 (127.3.222.3:34071), 5bc6d073b86141c8a5d3516577f787f2 (127.3.222.1:37061)
I20260928 17:50:27.717505 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f7662537924307bd651dbef193fd08" candidate_uuid: "38930024892b4ec7b142e1c08652a7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c034cde3ce64475cac13699dd7eeaf25"
I20260928 17:50:27.717594 22964 raft_consensus.cc:3037] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:27.718304 22964 raft_consensus.cc:2454] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38930024892b4ec7b142e1c08652a7d1 in term 1.
I20260928 17:50:27.718492 22768 leader_election.cc:302] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38930024892b4ec7b142e1c08652a7d1, c034cde3ce64475cac13699dd7eeaf25; no voters:
I20260928 17:50:27.718480 22702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f7662537924307bd651dbef193fd08" candidate_uuid: "38930024892b4ec7b142e1c08652a7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc6d073b86141c8a5d3516577f787f2"
I20260928 17:50:27.718546 22702 raft_consensus.cc:3037] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:27.718576 23030 raft_consensus.cc:2781] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:27.718727 23030 raft_consensus.cc:686] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 1 LEADER]: Becoming Leader. State: Replica: 38930024892b4ec7b142e1c08652a7d1, State: Running, Role: LEADER
I20260928 17:50:27.718811 23030 consensus_queue.cc:237] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } }
I20260928 17:50:27.719432 22702 raft_consensus.cc:2454] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38930024892b4ec7b142e1c08652a7d1 in term 1.
I20260928 17:50:27.719797 22559 catalog_manager.cc:5495] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38930024892b4ec7b142e1c08652a7d1 (127.3.222.2). New cstate: current_term: 1 leader_uuid: "38930024892b4ec7b142e1c08652a7d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38930024892b4ec7b142e1c08652a7d1" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 35285 } health_report { overall_health: HEALTHY } } }
I20260928 17:50:27.733880 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.731385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.3.222.2:35285)
0928 17:50:27.731432 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:27.731433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:27.731441 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:50:27.731514 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:27.731514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:27.731530 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:27.731589 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:27.731688 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.732147 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.732992 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:27.733177 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:27.733245 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:27.733500 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:27.733598 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:27.733689 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:27.733779 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 17:50:27.733795 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:50:27.735579 22964 raft_consensus.cc:1261] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [term 1 FOLLOWER]: Refusing update from remote peer 38930024892b4ec7b142e1c08652a7d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:27.735579 22702 raft_consensus.cc:1261] T e1f7662537924307bd651dbef193fd08 P 5bc6d073b86141c8a5d3516577f787f2 [term 1 FOLLOWER]: Refusing update from remote peer 38930024892b4ec7b142e1c08652a7d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:27.735901 23034 consensus_queue.cc:1035] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c034cde3ce64475cac13699dd7eeaf25" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:27.736022 23030 consensus_queue.cc:1035] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc6d073b86141c8a5d3516577f787f2" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:27.738097 23037 mvcc.cc:205] Tried to move back new op lower bound from 7334370622402248704 to 7334370622337306624. Current Snapshot: MvccSnapshot[applied={T|T < 7334370622402248704}]
I20260928 17:50:27.738731 23038 mvcc.cc:205] Tried to move back new op lower bound from 7334370622402248704 to 7334370622337306624. Current Snapshot: MvccSnapshot[applied={T|T < 7334370622402248704}]
I20260928 17:50:27.740203 23040 mvcc.cc:205] Tried to move back new op lower bound from 7334370622402248704 to 7334370622337306624. Current Snapshot: MvccSnapshot[applied={T|T < 7334370622402248704}]
I20260928 17:50:27.740326 22560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:27.740409 22560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:27.741115 22560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6d13db7bc7a47249befe4ade7ef293c: Sending DeleteTablet for 3 replicas of tablet e1f7662537924307bd651dbef193fd08
I20260928 17:50:27.741447 22944 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f7662537924307bd651dbef193fd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:27 UTC) from {username='slave'} at 127.0.0.1:52546
I20260928 17:50:27.741492 22813 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f7662537924307bd651dbef193fd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:27 UTC) from {username='slave'} at 127.0.0.1:49198
I20260928 17:50:27.741658 23047 tablet_replica.cc:330] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: stopping tablet replica
I20260928 17:50:27.741756 23047 raft_consensus.cc:2229] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:27.741797 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 22620
I20260928 17:50:27.744479 23047 raft_consensus.cc:2258] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:27.744974 23047 ts_tablet_manager.cc:1905] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 17:50:27.746803 22547 connection.cc:535] client connection to 127.3.222.1:37061 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 17:50:27.746883 22547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:50:27.747037 23048 tablet_replica.cc:330] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1: stopping tablet replica
I20260928 17:50:27.747125 23048 raft_consensus.cc:2229] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:50:27.747232 23048 raft_consensus.cc:2258] T e1f7662537924307bd651dbef193fd08 P 38930024892b4ec7b142e1c08652a7d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:27.747290 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.747138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:37061 (local address 127.0.0.1:52332)
0928 17:50:27.747192 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.747257 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 17:50:27.747367 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 22751
W20260928 17:50:27.747406 22547 catalog_manager.cc:4584] TS 5bc6d073b86141c8a5d3516577f787f2 (127.3.222.1:37061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1f7662537924307bd651dbef193fd08: Network error: Client connection negotiation failed: client connection to 127.3.222.1:37061: connect: Connection refused (error 111)
I20260928 17:50:27.748018 23047 ts_tablet_manager.cc:1918] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:27.748075 23047 log.cc:1196] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/e1f7662537924307bd651dbef193fd08
I20260928 17:50:27.748306 23047 ts_tablet_manager.cc:1939] T e1f7662537924307bd651dbef193fd08 P c034cde3ce64475cac13699dd7eeaf25: Deleting consensus metadata
I20260928 17:50:27.748677 22546 catalog_manager.cc:4841] TS c034cde3ce64475cac13699dd7eeaf25 (127.3.222.3:34071): tablet e1f7662537924307bd651dbef193fd08 (table test-table [id=b1b179fbb57f4bfbbe29f6a08b8a3f78]) successfully deleted
I20260928 17:50:27.751554 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 22882
I20260928 17:50:27.751611 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:27.751518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:35285 (local address 127.0.0.1:49212)
0928 17:50:27.751559 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:27.751582 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.2:35285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:50:27.755618 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 22529
2026-09-28T17:50:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (658 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:27 dist-test-slave-w3bm krb5kdc[23052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:27 dist-test-slave-w3bm krb5kdc[23052](info): set up 2 sockets
Sep 28 17:50:27 dist-test-slave-w3bm krb5kdc[23052](info): commencing operation
krb5kdc: starting...
W20260928 17:50:28.706698 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:28 dist-test-slave-w3bm krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:28.817868 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38449
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:38449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:28.894618 23065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:28.894814 23065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:28.894845 23065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:28.895961 23065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:28.896000 23065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:28.896018 23065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:28.896034 23065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:28.896049 23065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:28.897454 23065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:38449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:38449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:28.897675 23065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:28.897866 23065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:28.902313 23070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:28.902345 23071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:28.902320 23073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:28.902560 23065 server_base.cc:1034] running on GCE node
I20260928 17:50:28.902773 23065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:28.903021 23065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:28.904165 23065 hybrid_clock.cc:648] HybridClock initialized: now 1790617828904141 us; error 33 us; skew 500 ppm
Sep 28 17:50:28 dist-test-slave-w3bm krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617828, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:28.906445 23065 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:28.906845 23065 webserver.cc:460] Webserver started at http://127.3.222.62:36259/ using document root <none> and password file <none>
I20260928 17:50:28.907030 23065 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:28.907074 23065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:28.907194 23065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:28.908200 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "70d5589453314997a00c8399435adc5c"
format_stamp: "Formatted at 2026-09-28 17:50:28 on dist-test-slave-w3bm"
server_key: "83317224a7363115a119d8cf3a97232d"
server_key_iv: "76f052595aa7255acfcfab9caac1152c"
server_key_version: "encryptionkey@0"
I20260928 17:50:28.908516 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "70d5589453314997a00c8399435adc5c"
format_stamp: "Formatted at 2026-09-28 17:50:28 on dist-test-slave-w3bm"
server_key: "83317224a7363115a119d8cf3a97232d"
server_key_iv: "76f052595aa7255acfcfab9caac1152c"
server_key_version: "encryptionkey@0"
I20260928 17:50:28.909659 23065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:28.910286 23080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:28.910440 23065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:28.910516 23065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "70d5589453314997a00c8399435adc5c"
format_stamp: "Formatted at 2026-09-28 17:50:28 on dist-test-slave-w3bm"
server_key: "83317224a7363115a119d8cf3a97232d"
server_key_iv: "76f052595aa7255acfcfab9caac1152c"
server_key_version: "encryptionkey@0"
I20260928 17:50:28.910574 23065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:28.931068 23065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:28.933038 23065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:28.933166 23065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:28.936846 23065 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:38449
I20260928 17:50:28.936879 23132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:38449 every 8 connection(s)
I20260928 17:50:28.937175 23065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:28.937927 23133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:28.940135 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c: Bootstrap starting.
I20260928 17:50:28.940632 23133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:28.940860 23133 log.cc:824] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:28.941430 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c: No bootstrap required, opened a new log
I20260928 17:50:28.942075 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 23065
I20260928 17:50:28.942176 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:28.942365 3960 external_mini_cluster.cc:1428] Setting key a91b580e8d1c1b3f8b33f2e510bd0907
I20260928 17:50:28.943397 23133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5589453314997a00c8399435adc5c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38449 } }
I20260928 17:50:28.943550 23133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:28.943588 23133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d5589453314997a00c8399435adc5c, State: Initialized, Role: FOLLOWER
I20260928 17:50:28.943713 23133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5589453314997a00c8399435adc5c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38449 } }
I20260928 17:50:28.943791 23133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:28.943826 23133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:28.943873 23133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:28.944825 23133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5589453314997a00c8399435adc5c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38449 } }
I20260928 17:50:28.944926 23133 leader_election.cc:302] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70d5589453314997a00c8399435adc5c; no voters:
I20260928 17:50:28.945062 23133 leader_election.cc:288] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:28.945096 23138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:28.945276 23138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [term 1 LEADER]: Becoming Leader. State: Replica: 70d5589453314997a00c8399435adc5c, State: Running, Role: LEADER
I20260928 17:50:28.945295 23133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:28.945405 23138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5589453314997a00c8399435adc5c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38449 } }
I20260928 17:50:28.945834 23140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d5589453314997a00c8399435adc5c. Latest consensus state: current_term: 1 leader_uuid: "70d5589453314997a00c8399435adc5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5589453314997a00c8399435adc5c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38449 } } }
I20260928 17:50:28.945855 23139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70d5589453314997a00c8399435adc5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5589453314997a00c8399435adc5c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 38449 } } }
I20260928 17:50:28.945919 23140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:28.945933 23139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:28.946120 23146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:28.946564 23146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:28.948105 23146 catalog_manager.cc:1347] Generated new cluster ID: bb41644011ab4018b39d7f1543ccb2b7
I20260928 17:50:28.948146 23146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:50:28 dist-test-slave-w3bm krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:28.953229 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:28.942831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.3.222.62:38449)
0928 17:50:28.942972 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:28.942979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:28.943005 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:50:28.943358 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:28.943363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:28.943405 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:28.943519 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:28.944281 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:28.945346 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:28.946377 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:28.947037 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:28.947142 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:28.949764 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:28.949768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:28.949769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:28.949780 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:28.951662 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:28.952032 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:28.952035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:28.952035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:28.952162 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:28.952329 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:28.952330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:28.952330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:28.952629 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:28.952789 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:28.952859 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 17:50:28.953029 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260928 17:50:28.966507 23146 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:28.967029 23146 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:28.974018 23146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c: Generated new TSK 0
I20260928 17:50:28.974212 23146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
Entry for principal kudu/127.3.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab.
I20260928 17:50:28.998636 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38449
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:29.077176 23159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:29.077374 23159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:29.077395 23159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:29.078497 23159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:29.078536 23159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:29.078559 23159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:50:29.079982 23159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.1.keytab
--principal=kudu/127.3.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:29.080189 23159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:29.080400 23159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:29.085019 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.085080 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.085093 23167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:29.085230 23159 server_base.cc:1034] running on GCE node
I20260928 17:50:29.085515 23159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:29.085709 23159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:29.086876 23159 hybrid_clock.cc:648] HybridClock initialized: now 1790617829086874 us; error 38 us; skew 500 ppm
Sep 28 17:50:29 dist-test-slave-w3bm krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617829, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:29.089303 23159 init.cc:377] Logged in from keytab as kudu/127.3.222.1@KRBTEST.COM (short username kudu)
I20260928 17:50:29.089705 23159 webserver.cc:460] Webserver started at http://127.3.222.1:44405/ using document root <none> and password file <none>
I20260928 17:50:29.089926 23159 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:29.089973 23159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:29.090093 23159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:29.091084 23159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "2187526054e44a7c818911d6046a0253"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "1ce12e58efac392bb4b71a2bd847f81b"
server_key_iv: "397a8f1d35c582908b71e03b85d59846"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.091418 23159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "2187526054e44a7c818911d6046a0253"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "1ce12e58efac392bb4b71a2bd847f81b"
server_key_iv: "397a8f1d35c582908b71e03b85d59846"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.092597 23159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:29.093284 23174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.093441 23159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:29.093516 23159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "2187526054e44a7c818911d6046a0253"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "1ce12e58efac392bb4b71a2bd847f81b"
server_key_iv: "397a8f1d35c582908b71e03b85d59846"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.093580 23159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:29.115830 23159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:29.117545 23159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:29.117659 23159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:29.117882 23159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:29.118160 23159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:29.118192 23159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.118213 23159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:29.118237 23159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.123898 23159 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:33075
I20260928 17:50:29.123950 23287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:33075 every 8 connection(s)
I20260928 17:50:29.124231 23159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:50:29 dist-test-slave-w3bm krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617829, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.1@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:29.132575 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.124903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:37491 (local address 127.3.222.62:38449)
0928 17:50:29.124944 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.124946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.125130 (+ 184us) server_negotiation.cc:407] Connection header received
0928 17:50:29.125625 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.125627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.125643 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.125692 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:29.126583 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.127089 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.128078 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.128287 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.128342 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.130279 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:29.130281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:29.130281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:29.130288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:29.131433 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:29.131826 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:29.131827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:29.131827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:29.131871 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:29.132063 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:29.132064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:29.132064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:29.132159 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:29.132203 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.132436 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.132508 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:50:29.132715 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.124859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38449 (local address 127.3.222.1:37491)
0928 17:50:29.125019 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.125053 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:29.125536 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:29.125762 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:29.125766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:29.126311 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:29.126495 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.126499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.127166 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.127166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.128000 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.128002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.128354 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.128355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.128481 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.128891 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:29.128899 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:29.130183 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:29.131519 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:29.131531 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:29.131532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:29.131730 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:29.131922 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:29.131923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:29.131923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:29.132019 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:29.132208 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:29.132212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:29.132332 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 17:50:29.132419 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 17:50:29.132532 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260928 17:50:29.133162 23288 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38449
I20260928 17:50:29.133241 23288 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:29.133379 23288 heartbeater.cc:511] Master 127.3.222.62:38449 requested a full tablet report, sending...
I20260928 17:50:29.133718 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 23159
I20260928 17:50:29.133824 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:50:29.133899 23097 ts_manager.cc:194] Registered new tserver with Master: 2187526054e44a7c818911d6046a0253 (127.3.222.1:33075)
I20260928 17:50:29.134019 3960 external_mini_cluster.cc:1428] Setting key 36cb0472c58613019e9d3001f26dd231
I20260928 17:50:29.134590 23097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.1@KRBTEST.COM'} at 127.3.222.1:37491
WARNING: no policy specified for kudu/127.3.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
Entry for principal kudu/127.3.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab.
I20260928 17:50:29.159518 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38449
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:29.237294 23293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:29.237483 23293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:29.237504 23293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:29.238621 23293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:29.238660 23293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:29.238684 23293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:50:29.240119 23293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.2.keytab
--principal=kudu/127.3.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:29.240330 23293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:29.240527 23293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:29.245386 23298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.245365 23299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:29.245400 23293 server_base.cc:1034] running on GCE node
W20260928 17:50:29.245359 23301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:29.245755 23293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:29.245957 23293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:29.247094 23293 hybrid_clock.cc:648] HybridClock initialized: now 1790617829247077 us; error 41 us; skew 500 ppm
Sep 28 17:50:29 dist-test-slave-w3bm krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617829, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:29.249433 23293 init.cc:377] Logged in from keytab as kudu/127.3.222.2@KRBTEST.COM (short username kudu)
I20260928 17:50:29.249820 23293 webserver.cc:460] Webserver started at http://127.3.222.2:42793/ using document root <none> and password file <none>
I20260928 17:50:29.249994 23293 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:29.250032 23293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:29.250113 23293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:29.251091 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "fe595320951244fa83f044d14e62d493"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "b60498b125cb2a0d6c9872dfb2fb4086"
server_key_iv: "d3958682aacfb23f4a0f7453f5aa7d4b"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.251412 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "fe595320951244fa83f044d14e62d493"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "b60498b125cb2a0d6c9872dfb2fb4086"
server_key_iv: "d3958682aacfb23f4a0f7453f5aa7d4b"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.252629 23293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:29.253311 23308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.253477 23293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:29.253569 23293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "fe595320951244fa83f044d14e62d493"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "b60498b125cb2a0d6c9872dfb2fb4086"
server_key_iv: "d3958682aacfb23f4a0f7453f5aa7d4b"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.253639 23293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:29.263159 23293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:29.264962 23293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:29.265071 23293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:29.265261 23293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:29.265543 23293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:29.265575 23293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.265595 23293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:29.265619 23293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.270853 23293 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:46839
I20260928 17:50:29.270954 23421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:46839 every 8 connection(s)
I20260928 17:50:29.271166 23293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:50:29.273938 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 23293
I20260928 17:50:29.274039 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:50:29.274204 3960 external_mini_cluster.cc:1428] Setting key 9c2eb29b0fe1002746b258f598d16aac
Sep 28 17:50:29 dist-test-slave-w3bm krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617829, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.2@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:29.280483 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.271869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:33199 (local address 127.3.222.62:38449)
0928 17:50:29.271930 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.271932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.272131 (+ 199us) server_negotiation.cc:407] Connection header received
0928 17:50:29.272614 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.272615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.272624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.272672 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:29.273555 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.274146 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.275479 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.275668 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.275724 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.277768 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:29.277769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:29.277769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:29.277775 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:29.278997 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:29.279472 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:29.279473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:29.279474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:29.279526 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:29.279837 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:29.279838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:29.279839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:29.279966 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:29.280028 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.280346 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.280419 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:50:29.280752 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.271842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38449 (local address 127.3.222.2:33199)
0928 17:50:29.272018 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.272057 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:29.272536 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:29.272725 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:29.272728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:29.273272 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:29.273455 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.273460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.274216 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.274217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.275406 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.275410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.275738 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.275739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.275880 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.276376 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:29.276384 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:29.277688 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:29.279072 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:29.279079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:29.279081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:29.279404 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:29.279583 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:29.279584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:29.279584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:29.279716 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:29.280033 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:29.280040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:29.280202 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 17:50:29.280327 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 17:50:29.280487 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260928 17:50:29.281236 23422 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38449
I20260928 17:50:29.281340 23422 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:29.281523 23422 heartbeater.cc:511] Master 127.3.222.62:38449 requested a full tablet report, sending...
I20260928 17:50:29.281936 23097 ts_manager.cc:194] Registered new tserver with Master: fe595320951244fa83f044d14e62d493 (127.3.222.2:46839)
I20260928 17:50:29.282285 23097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.2@KRBTEST.COM'} at 127.3.222.2:33199
WARNING: no policy specified for kudu/127.3.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
Entry for principal kudu/127.3.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab.
I20260928 17:50:29.298444 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38449
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:29.375846 23427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:29.376047 23427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:29.376077 23427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:29.377163 23427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:29.377200 23427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:29.377234 23427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:50:29.378626 23427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.3.keytab
--principal=kudu/127.3.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:38449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:29.378844 23427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:29.379035 23427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:29.383518 23435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.383528 23432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.383570 23433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:29.383774 23427 server_base.cc:1034] running on GCE node
I20260928 17:50:29.384011 23427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:29.384217 23427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:29.385347 23427 hybrid_clock.cc:648] HybridClock initialized: now 1790617829385334 us; error 31 us; skew 500 ppm
Sep 28 17:50:29 dist-test-slave-w3bm krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617829, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:29.387645 23427 init.cc:377] Logged in from keytab as kudu/127.3.222.3@KRBTEST.COM (short username kudu)
I20260928 17:50:29.388087 23427 webserver.cc:460] Webserver started at http://127.3.222.3:46471/ using document root <none> and password file <none>
I20260928 17:50:29.388273 23427 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:29.388316 23427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:29.388438 23427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:29.389393 23427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "006320bf4eed4d4c925e0ebfe8dda857"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "f0def2806468f83121513d599d43a8d7"
server_key_iv: "92f404eab705ff1772b33ae08b9586a4"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.389701 23427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "006320bf4eed4d4c925e0ebfe8dda857"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "f0def2806468f83121513d599d43a8d7"
server_key_iv: "92f404eab705ff1772b33ae08b9586a4"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.390852 23427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:29.391515 23442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.391652 23427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.391728 23427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "006320bf4eed4d4c925e0ebfe8dda857"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "f0def2806468f83121513d599d43a8d7"
server_key_iv: "92f404eab705ff1772b33ae08b9586a4"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.391793 23427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:29.417966 23427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:29.420025 23427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:29.420156 23427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:29.420367 23427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:29.420639 23427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:29.420670 23427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.420698 23427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:29.420722 23427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.425940 23427 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:43725
I20260928 17:50:29.425981 23555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:43725 every 8 connection(s)
I20260928 17:50:29.426268 23427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:50:29 dist-test-slave-w3bm krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617829, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.3@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:29.432910 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 23427
I20260928 17:50:29.432999 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:50:29.433200 3960 external_mini_cluster.cc:1428] Setting key daf4d8aa4e42d21b0b7b1773b76982fd
I20260928 17:50:29.434981 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.426946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:47225 (local address 127.3.222.62:38449)
0928 17:50:29.427000 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.427002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.427127 (+ 125us) server_negotiation.cc:407] Connection header received
0928 17:50:29.427631 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.427632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.427643 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.427697 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:29.428559 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.429093 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.430127 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.430320 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.430377 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.432345 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:29.432346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:29.432347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:29.432353 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:29.433706 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:29.434148 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:29.434149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:29.434149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:29.434207 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:29.434416 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:29.434417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:29.434417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:29.434526 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:29.434580 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.434819 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.434894 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:50:29.435148 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.426875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:38449 (local address 127.3.222.3:47225)
0928 17:50:29.427015 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.427055 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:29.427523 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:29.427749 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:29.427752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:29.428276 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:29.428462 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.428467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.429194 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.429194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.430039 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.430042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.430385 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.430385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.430481 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.430913 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:29.430921 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:29.432258 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:29.433784 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:29.433834 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:29.433836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:29.434085 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:29.434258 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:29.434258 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:29.434259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:29.434359 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:29.434576 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:29.434582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:29.434708 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 17:50:29.434799 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 17:50:29.434948 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260928 17:50:29.435539 23556 heartbeater.cc:347] Connected to a master server at 127.3.222.62:38449
I20260928 17:50:29.435629 23556 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:29.435791 23556 heartbeater.cc:511] Master 127.3.222.62:38449 requested a full tablet report, sending...
I20260928 17:50:29.436095 23097 ts_manager.cc:194] Registered new tserver with Master: 006320bf4eed4d4c925e0ebfe8dda857 (127.3.222.3:43725)
I20260928 17:50:29.436458 23097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.222.3@KRBTEST.COM'} at 127.3.222.3:47225
I20260928 17:50:29.444729 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:50:29.451613 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.446183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.3.222.62:38449)
0928 17:50:29.446224 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.446226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.446304 (+ 78us) server_negotiation.cc:407] Connection header received
0928 17:50:29.446351 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.446351 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.446359 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.446401 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:29.446947 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.447471 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.448165 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.448361 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.448417 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.449464 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:29.449465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:29.449465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:29.449471 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:29.450703 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:29.450997 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:29.450997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:29.450998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:29.451041 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:29.451163 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:29.451163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:29.451163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:29.451333 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:29.451406 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.451499 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.451554 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:50:29.452899 23097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:50:29.454490 23097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:50:29.460191 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.456443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:33075 (local address 127.0.0.1:54264)
0928 17:50:29.456549 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.456582 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:29.456643 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:29.456924 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:29.456927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:29.456929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:29.456992 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.456993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.458078 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.458079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.459016 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.459018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.459909 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.459910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.459988 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.460002 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:50:29.460070 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:50:29.460120 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260928 17:50:29.460207 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.456552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54264 (local address 127.3.222.1:33075)
0928 17:50:29.456750 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.456756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.456768 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:50:29.456807 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.456809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.456825 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.456910 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:29.457058 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.457988 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.459114 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.459840 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.459904 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.459962 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.460087 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.460127 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260928 17:50:29.460821 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.456625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:43725 (local address 127.0.0.1:52010)
0928 17:50:29.457019 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.457030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:29.457071 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:29.457335 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:29.457336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:29.457338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:29.457393 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.457395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.458668 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.458670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.459652 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.459655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.460591 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.460592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.460658 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.460662 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:29.460713 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:50:29.460752 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":27,"thread_start_us":44,"threads_started":1}
I20260928 17:50:29.460841 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.456730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.3.222.3:43725)
0928 17:50:29.457129 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.457133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.457142 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:50:29.457176 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.457178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.457192 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.457259 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:29.457465 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.458572 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.459724 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.460508 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.460571 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.460625 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.460729 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.460767 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":158,"threads_started":1}
I20260928 17:50:29.460985 23222 tablet_service.cc:1467] Processing CreateTablet for tablet b956cca4302c4dd0843c604168c3ab5f (DEFAULT_TABLE table=test-table [id=59fddc135d814da1b8b2d854584fbc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:29.461274 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.456749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:46839 (local address 127.0.0.1:45856)
0928 17:50:29.457058 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.457068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:29.457148 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:29.457398 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:29.457399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:29.457401 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:29.457427 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.457429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.458674 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.458676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.459717 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.459720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.461056 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.461057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.461119 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.461123 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:29.461167 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:50:29.461203 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":48,"thread_start_us":48,"threads_started":1}
I20260928 17:50:29.461338 23222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b956cca4302c4dd0843c604168c3ab5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:29.461477 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.456853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.3.222.2:46839)
0928 17:50:29.457014 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.457018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.457122 (+ 104us) server_negotiation.cc:407] Connection header received
0928 17:50:29.457245 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.457248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.457261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.457327 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:29.457494 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.458566 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.459807 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.460762 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.460852 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.461024 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.461365 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.461405 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
I20260928 17:50:29.461581 23490 tablet_service.cc:1467] Processing CreateTablet for tablet b956cca4302c4dd0843c604168c3ab5f (DEFAULT_TABLE table=test-table [id=59fddc135d814da1b8b2d854584fbc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:29.461884 23490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b956cca4302c4dd0843c604168c3ab5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:29.462083 23356 tablet_service.cc:1467] Processing CreateTablet for tablet b956cca4302c4dd0843c604168c3ab5f (DEFAULT_TABLE table=test-table [id=59fddc135d814da1b8b2d854584fbc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:29.462399 23356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b956cca4302c4dd0843c604168c3ab5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:29.464200 23570 tablet_bootstrap.cc:492] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: Bootstrap starting.
I20260928 17:50:29.464774 23570 tablet_bootstrap.cc:654] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:29.464820 23571 tablet_bootstrap.cc:492] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253: Bootstrap starting.
I20260928 17:50:29.465214 23570 log.cc:824] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:29.465440 23571 tablet_bootstrap.cc:654] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:29.465830 23572 tablet_bootstrap.cc:492] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493: Bootstrap starting.
I20260928 17:50:29.465912 23570 tablet_bootstrap.cc:492] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: No bootstrap required, opened a new log
I20260928 17:50:29.465920 23571 log.cc:824] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:29.465966 23570 ts_tablet_manager.cc:1397] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:29.466434 23572 tablet_bootstrap.cc:654] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:29.466534 23571 tablet_bootstrap.cc:492] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253: No bootstrap required, opened a new log
I20260928 17:50:29.466595 23571 ts_tablet_manager.cc:1397] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:29.466977 23572 log.cc:824] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:29.467712 23572 tablet_bootstrap.cc:492] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493: No bootstrap required, opened a new log
I20260928 17:50:29.467780 23572 ts_tablet_manager.cc:1397] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:29.467816 23570 raft_consensus.cc:348] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } }
I20260928 17:50:29.467988 23570 raft_consensus.cc:374] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:29.468027 23570 raft_consensus.cc:729] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006320bf4eed4d4c925e0ebfe8dda857, State: Initialized, Role: FOLLOWER
I20260928 17:50:29.468132 23570 consensus_queue.cc:260] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } }
I20260928 17:50:29.468215 23571 raft_consensus.cc:348] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } }
I20260928 17:50:29.468380 23571 raft_consensus.cc:374] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:29.468397 23570 ts_tablet_manager.cc:1428] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 17:50:29.468420 23571 raft_consensus.cc:729] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2187526054e44a7c818911d6046a0253, State: Initialized, Role: FOLLOWER
I20260928 17:50:29.468515 23556 heartbeater.cc:503] Master 127.3.222.62:38449 was elected leader, sending a full tablet report...
I20260928 17:50:29.468546 23571 consensus_queue.cc:260] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } }
I20260928 17:50:29.468811 23571 ts_tablet_manager.cc:1428] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:50:29.468832 23288 heartbeater.cc:503] Master 127.3.222.62:38449 was elected leader, sending a full tablet report...
I20260928 17:50:29.469763 23572 raft_consensus.cc:348] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } }
I20260928 17:50:29.469924 23572 raft_consensus.cc:374] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:29.469967 23572 raft_consensus.cc:729] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe595320951244fa83f044d14e62d493, State: Initialized, Role: FOLLOWER
I20260928 17:50:29.470080 23572 consensus_queue.cc:260] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } }
I20260928 17:50:29.470280 23572 ts_tablet_manager.cc:1428] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:50:29.470312 23422 heartbeater.cc:503] Master 127.3.222.62:38449 was elected leader, sending a full tablet report...
W20260928 17:50:29.521708 23423 tablet.cc:2369] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:50:29.532612 23576 raft_consensus.cc:482] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:50:29.532728 23576 raft_consensus.cc:504] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } }
I20260928 17:50:29.533069 23576 leader_election.cc:288] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2187526054e44a7c818911d6046a0253 (127.3.222.1:33075), fe595320951244fa83f044d14e62d493 (127.3.222.2:46839)
I20260928 17:50:29.535841 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.533299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:57717 (local address 127.3.222.1:33075)
0928 17:50:29.533346 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.533348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.533357 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:50:29.533415 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.533416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.533428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.533478 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:29.533623 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.534165 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.535151 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.535532 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.535583 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.535591 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.535735 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.535774 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:50:29.535869 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.533414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:37709 (local address 127.3.222.2:46839)
0928 17:50:29.533455 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.533457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.533539 (+ 82us) server_negotiation.cc:407] Connection header received
0928 17:50:29.533580 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.533581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.533593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.533640 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:29.533791 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.534242 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.535156 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.535581 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.535645 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.535656 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.535774 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.535811 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:50:29.535861 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.533219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:33075 (local address 127.3.222.3:57717)
0928 17:50:29.533276 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.533290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:29.533361 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:29.533486 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:29.533487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:29.533489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:29.533559 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.533561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.534231 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.534232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.535067 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.535070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.535595 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.535596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.535673 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.535677 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:29.535724 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 17:50:29.535760 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":68}
I20260928 17:50:29.535884 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.533313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:46839 (local address 127.3.222.3:37709)
0928 17:50:29.533479 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.533487 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:29.533531 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:29.533649 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:29.533649 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:29.533651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:29.533722 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.533724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.534312 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.534313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.535069 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.535071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.535654 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.535655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.535706 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.535710 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:29.535761 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:50:29.535793 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260928 17:50:29.536279 23242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b956cca4302c4dd0843c604168c3ab5f" candidate_uuid: "006320bf4eed4d4c925e0ebfe8dda857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2187526054e44a7c818911d6046a0253" is_pre_election: true
I20260928 17:50:29.536438 23242 raft_consensus.cc:2454] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 006320bf4eed4d4c925e0ebfe8dda857 in term 0.
I20260928 17:50:29.536399 23376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b956cca4302c4dd0843c604168c3ab5f" candidate_uuid: "006320bf4eed4d4c925e0ebfe8dda857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe595320951244fa83f044d14e62d493" is_pre_election: true
I20260928 17:50:29.536538 23376 raft_consensus.cc:2454] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 006320bf4eed4d4c925e0ebfe8dda857 in term 0.
I20260928 17:50:29.536639 23446 leader_election.cc:302] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 006320bf4eed4d4c925e0ebfe8dda857, 2187526054e44a7c818911d6046a0253; no voters:
I20260928 17:50:29.536779 23576 raft_consensus.cc:2781] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:50:29.536839 23576 raft_consensus.cc:482] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:50:29.536854 23576 raft_consensus.cc:3037] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:29.537496 23576 raft_consensus.cc:504] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } }
I20260928 17:50:29.537596 23576 leader_election.cc:288] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [CANDIDATE]: Term 1 election: Requested vote from peers 2187526054e44a7c818911d6046a0253 (127.3.222.1:33075), fe595320951244fa83f044d14e62d493 (127.3.222.2:46839)
I20260928 17:50:29.537739 23242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b956cca4302c4dd0843c604168c3ab5f" candidate_uuid: "006320bf4eed4d4c925e0ebfe8dda857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2187526054e44a7c818911d6046a0253"
I20260928 17:50:29.537745 23376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b956cca4302c4dd0843c604168c3ab5f" candidate_uuid: "006320bf4eed4d4c925e0ebfe8dda857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe595320951244fa83f044d14e62d493"
I20260928 17:50:29.537807 23376 raft_consensus.cc:3037] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:29.537807 23242 raft_consensus.cc:3037] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:29.538625 23376 raft_consensus.cc:2454] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 006320bf4eed4d4c925e0ebfe8dda857 in term 1.
I20260928 17:50:29.538625 23242 raft_consensus.cc:2454] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 006320bf4eed4d4c925e0ebfe8dda857 in term 1.
I20260928 17:50:29.538796 23446 leader_election.cc:302] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 006320bf4eed4d4c925e0ebfe8dda857, fe595320951244fa83f044d14e62d493; no voters:
I20260928 17:50:29.538877 23576 raft_consensus.cc:2781] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:29.539005 23576 raft_consensus.cc:686] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 1 LEADER]: Becoming Leader. State: Replica: 006320bf4eed4d4c925e0ebfe8dda857, State: Running, Role: LEADER
I20260928 17:50:29.539080 23576 consensus_queue.cc:237] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } }
I20260928 17:50:29.539916 23097 catalog_manager.cc:5495] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 reported cstate change: term changed from 0 to 1, leader changed from <none> to 006320bf4eed4d4c925e0ebfe8dda857 (127.3.222.3). New cstate: current_term: 1 leader_uuid: "006320bf4eed4d4c925e0ebfe8dda857" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "006320bf4eed4d4c925e0ebfe8dda857" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20260928 17:50:29.552241 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.550002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.3.222.3:43725)
0928 17:50:29.550067 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.550069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.550093 (+ 24us) server_negotiation.cc:407] Connection header received
0928 17:50:29.550123 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.550124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.550143 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.550199 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:29.550314 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.550783 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.551563 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.551758 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.551811 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.551957 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:29.552051 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:29.552120 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.552154 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.552173 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:50:29.553846 23376 raft_consensus.cc:1261] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [term 1 FOLLOWER]: Refusing update from remote peer 006320bf4eed4d4c925e0ebfe8dda857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:29.553867 23242 raft_consensus.cc:1261] T b956cca4302c4dd0843c604168c3ab5f P 2187526054e44a7c818911d6046a0253 [term 1 FOLLOWER]: Refusing update from remote peer 006320bf4eed4d4c925e0ebfe8dda857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:29.554173 23576 consensus_queue.cc:1035] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe595320951244fa83f044d14e62d493" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:29.554303 23580 consensus_queue.cc:1035] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2187526054e44a7c818911d6046a0253" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:29.556447 23583 mvcc.cc:205] Tried to move back new op lower bound from 7334370629850116096 to 7334370629792931840. Current Snapshot: MvccSnapshot[applied={T|T < 7334370629850116096}]
I20260928 17:50:29.557087 23585 mvcc.cc:205] Tried to move back new op lower bound from 7334370629850116096 to 7334370629792931840. Current Snapshot: MvccSnapshot[applied={T|T < 7334370629850116096}]
I20260928 17:50:29.558758 23097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:29.558844 23097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:29.559561 23097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70d5589453314997a00c8399435adc5c: Sending DeleteTablet for 3 replicas of tablet b956cca4302c4dd0843c604168c3ab5f
I20260928 17:50:29.559849 23222 tablet_service.cc:1514] Processing DeleteTablet for tablet b956cca4302c4dd0843c604168c3ab5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:29 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:54264
I20260928 17:50:29.559904 23356 tablet_service.cc:1514] Processing DeleteTablet for tablet b956cca4302c4dd0843c604168c3ab5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:29 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:45856
I20260928 17:50:29.559942 23490 tablet_service.cc:1514] Processing DeleteTablet for tablet b956cca4302c4dd0843c604168c3ab5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:29 UTC) from {username='kudu', principal='kudu/127.3.222.62@KRBTEST.COM'} at 127.0.0.1:52010
I20260928 17:50:29.560201 23595 tablet_replica.cc:330] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: stopping tablet replica
I20260928 17:50:29.560282 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 23159
I20260928 17:50:29.560292 23595 raft_consensus.cc:2229] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:50:29.560408 23595 raft_consensus.cc:2258] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:29.561064 23595 ts_tablet_manager.cc:1905] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:50:29.561959 23594 tablet_replica.cc:330] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493: stopping tablet replica
I20260928 17:50:29.562045 23594 raft_consensus.cc:2229] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:29.562134 23594 raft_consensus.cc:2258] T b956cca4302c4dd0843c604168c3ab5f P fe595320951244fa83f044d14e62d493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:29.562765 23595 ts_tablet_manager.cc:1918] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:29.562839 23595 log.cc:1196] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/b956cca4302c4dd0843c604168c3ab5f
I20260928 17:50:29.563077 23595 ts_tablet_manager.cc:1939] T b956cca4302c4dd0843c604168c3ab5f P 006320bf4eed4d4c925e0ebfe8dda857: Deleting consensus metadata
I20260928 17:50:29.563452 23084 catalog_manager.cc:4841] TS 006320bf4eed4d4c925e0ebfe8dda857 (127.3.222.3:43725): tablet b956cca4302c4dd0843c604168c3ab5f (table test-table [id=59fddc135d814da1b8b2d854584fbc65]) successfully deleted
W20260928 17:50:29.565491 23084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:33075 (error 108)
I20260928 17:50:29.565800 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 23293
I20260928 17:50:29.565863 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.565766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:33075 (local address 127.0.0.1:54278)
0928 17:50:29.565802 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.565832 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:33075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 17:50:29.566566 23084 catalog_manager.cc:4584] TS 2187526054e44a7c818911d6046a0253 (127.3.222.1:33075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b956cca4302c4dd0843c604168c3ab5f: Network error: Client connection negotiation failed: client connection to 127.3.222.1:33075: connect: Connection refused (error 111)
W20260928 17:50:29.572264 23567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:29.572153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:46839 (local address 127.0.0.1:45872)
0928 17:50:29.572195 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.572223 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.2:46839: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:50:29.572499 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 23427
I20260928 17:50:29.576303 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 23065
2026-09-28T17:50:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1821 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T17:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:29Z Disabled control of system clock
I20260928 17:50:29.601635 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:42329
--webserver_interface=127.3.222.62
--webserver_port=0
--builtin_ntp_servers=127.3.222.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:29.677429 23600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:29.677630 23600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:29.677649 23600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:29.678763 23600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:29.678802 23600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:29.678812 23600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:29.678824 23600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:29.678838 23600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:29.680168 23600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:42329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:29.680387 23600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:29.680645 23600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:29.685321 23605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.685335 23606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:29.685391 23600 server_base.cc:1034] running on GCE node
W20260928 17:50:29.685336 23608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:29.685737 23600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:29.685994 23600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:29.687125 23600 hybrid_clock.cc:648] HybridClock initialized: now 1790617829687116 us; error 26 us; skew 500 ppm
I20260928 17:50:29.688218 23600 webserver.cc:460] Webserver started at http://127.3.222.62:39793/ using document root <none> and password file <none>
I20260928 17:50:29.688452 23600 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:29.688495 23600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:29.688616 23600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:29.689626 23600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "26c8cd598e074af99ec751648c5420af"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "e758fd6a2a3444356410fe1050b69d7d"
server_key_iv: "6fef06979c6e884c20aa89b98178dc56"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.689944 23600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "26c8cd598e074af99ec751648c5420af"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "e758fd6a2a3444356410fe1050b69d7d"
server_key_iv: "6fef06979c6e884c20aa89b98178dc56"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.691092 23600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:29.691781 23614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.691917 23600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.691995 23600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "26c8cd598e074af99ec751648c5420af"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "e758fd6a2a3444356410fe1050b69d7d"
server_key_iv: "6fef06979c6e884c20aa89b98178dc56"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.692060 23600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:29.717823 23600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:29.718134 23600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:29.718261 23600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:29.721841 23600 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:42329
I20260928 17:50:29.721871 23666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:42329 every 8 connection(s)
I20260928 17:50:29.722172 23600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:29.723090 23667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:29.725267 23667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af: Bootstrap starting.
I20260928 17:50:29.725409 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 23600
I20260928 17:50:29.725497 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:29.725654 3960 external_mini_cluster.cc:1428] Setting key cd72d740001e6e1f4e3ad43a7a9cb757
I20260928 17:50:29.725852 23667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:29.726171 23667 log.cc:824] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:29.726840 23667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af: No bootstrap required, opened a new log
I20260928 17:50:29.729028 23667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c8cd598e074af99ec751648c5420af" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 42329 } }
I20260928 17:50:29.729154 23667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:29.729174 23667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c8cd598e074af99ec751648c5420af, State: Initialized, Role: FOLLOWER
I20260928 17:50:29.729276 23667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c8cd598e074af99ec751648c5420af" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 42329 } }
I20260928 17:50:29.729342 23667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:29.729370 23667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:29.729414 23667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:29.730017 23667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c8cd598e074af99ec751648c5420af" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 42329 } }
I20260928 17:50:29.730119 23667 leader_election.cc:302] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26c8cd598e074af99ec751648c5420af; no voters:
I20260928 17:50:29.730270 23667 leader_election.cc:288] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:29.730420 23673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:29.730521 23667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:29.730546 23673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [term 1 LEADER]: Becoming Leader. State: Replica: 26c8cd598e074af99ec751648c5420af, State: Running, Role: LEADER
I20260928 17:50:29.730624 23673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c8cd598e074af99ec751648c5420af" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 42329 } }
I20260928 17:50:29.730970 23674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c8cd598e074af99ec751648c5420af. Latest consensus state: current_term: 1 leader_uuid: "26c8cd598e074af99ec751648c5420af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c8cd598e074af99ec751648c5420af" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 42329 } } }
I20260928 17:50:29.731047 23674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:29.731608 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.726094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.3.222.62:42329)
0928 17:50:29.726286 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.726295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.726333 (+ 38us) server_negotiation.cc:407] Connection header received
0928 17:50:29.726812 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.726821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.726899 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.727038 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:29.727198 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.728575 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.729365 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.729963 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.730084 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.730718 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:29.730722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:29.730722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:29.730909 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:29.730953 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:29.730967 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:29.731027 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.731328 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.731405 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260928 17:50:29.731894 23672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26c8cd598e074af99ec751648c5420af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c8cd598e074af99ec751648c5420af" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 42329 } } }
I20260928 17:50:29.732239 23672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af [sys.catalog]: This master's current role is: LEADER
W20260928 17:50:29.732281 23688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:29.732326 23688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:29.732365 23680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:29.732510 23680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:29.733769 23680 catalog_manager.cc:1347] Generated new cluster ID: 86fced558edf4b6d84bc1b947655ba45
I20260928 17:50:29.733832 23680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:29.742501 23680 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:29.743140 23680 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:29.759238 23680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af: Generated new TSK 0
I20260928 17:50:29.759442 23680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:29.768069 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.1:0
--local_ip_for_outbound_sockets=127.3.222.1
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:42329
--builtin_ntp_servers=127.3.222.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:29.846634 23691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:29.846822 23691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:29.846859 23691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:29.848004 23691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:29.848043 23691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:29.848075 23691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.1
I20260928 17:50:29.849375 23691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.222.1
--webserver_port=0
--tserver_master_addrs=127.3.222.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.1
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:29.849592 23691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:29.849859 23691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:50:29.854954 23691 server_base.cc:1034] running on GCE node
W20260928 17:50:29.854900 23696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.855057 23697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.854900 23699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:29.855312 23691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:29.855543 23691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:29.856719 23691 hybrid_clock.cc:648] HybridClock initialized: now 1790617829856703 us; error 27 us; skew 500 ppm
I20260928 17:50:29.857955 23691 webserver.cc:460] Webserver started at http://127.3.222.1:38935/ using document root <none> and password file <none>
I20260928 17:50:29.858191 23691 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:29.858237 23691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:29.858366 23691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:29.859449 23691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/data/instance:
uuid: "7c4ad15a615a4bd38781ded42503e45e"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "1912f5d65152be381325f45655d040e7"
server_key_iv: "b5e34b388f651ffa3bee6973d1fcc693"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.859756 23691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance:
uuid: "7c4ad15a615a4bd38781ded42503e45e"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "1912f5d65152be381325f45655d040e7"
server_key_iv: "b5e34b388f651ffa3bee6973d1fcc693"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.860972 23691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:29.861689 23705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.861871 23691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:29.861949 23691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/wal
uuid: "7c4ad15a615a4bd38781ded42503e45e"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "1912f5d65152be381325f45655d040e7"
server_key_iv: "b5e34b388f651ffa3bee6973d1fcc693"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.862008 23691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:29.869571 23691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:29.869827 23691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:29.869938 23691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:29.870122 23691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:29.870391 23691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:29.870424 23691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.870450 23691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:29.870473 23691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.875865 23691 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.1:32895
I20260928 17:50:29.875900 23818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.1:32895 every 8 connection(s)
I20260928 17:50:29.876186 23691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/data/info.pb
I20260928 17:50:29.880875 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.876830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:33775 (local address 127.3.222.62:42329)
0928 17:50:29.876875 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:29.876876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:29.877067 (+ 191us) server_negotiation.cc:407] Connection header received
0928 17:50:29.877559 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:29.877560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:29.877572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:29.877622 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:29.878238 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.878728 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.879701 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.879902 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.879955 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.880627 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:29.880629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:29.880629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:29.880633 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:29.880648 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:29.880652 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:29.880695 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:29.880765 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 17:50:29.880790 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:50:29.881052 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:29.876802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:42329 (local address 127.3.222.1:33775)
0928 17:50:29.876962 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:29.876994 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:29.877470 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:29.877687 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:29.877691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:29.877701 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:29.878158 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.878163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.878808 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.878809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.879627 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:29.879630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:29.879969 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:29.879970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:29.880083 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:29.880510 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:50:29.880510 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:29.880536 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:50:29.880557 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:29.880701 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:29.880706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:29.880709 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:29.880764 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 17:50:29.880856 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260928 17:50:29.881503 23819 heartbeater.cc:347] Connected to a master server at 127.3.222.62:42329
I20260928 17:50:29.881587 23819 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:29.881752 23819 heartbeater.cc:511] Master 127.3.222.62:42329 requested a full tablet report, sending...
I20260928 17:50:29.882243 23631 ts_manager.cc:194] Registered new tserver with Master: 7c4ad15a615a4bd38781ded42503e45e (127.3.222.1:32895)
I20260928 17:50:29.882500 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 23691
I20260928 17:50:29.882592 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-0/wal/instance
I20260928 17:50:29.882750 3960 external_mini_cluster.cc:1428] Setting key 3338dffc7b789412390fde7c7ffa6acd
I20260928 17:50:29.882927 23631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.1:33775
I20260928 17:50:29.883761 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.2:0
--local_ip_for_outbound_sockets=127.3.222.2
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:42329
--builtin_ntp_servers=127.3.222.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:29.960479 23822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:29.960681 23822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:29.960712 23822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:29.961829 23822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:29.961874 23822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:29.961905 23822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.2
I20260928 17:50:29.963194 23822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.222.2
--webserver_port=0
--tserver_master_addrs=127.3.222.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.2
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:29.963444 23822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:29.963699 23822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:29.968376 23828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.968492 23830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:29.968539 23827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:29.968595 23822 server_base.cc:1034] running on GCE node
I20260928 17:50:29.968788 23822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:29.969017 23822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:29.970153 23822 hybrid_clock.cc:648] HybridClock initialized: now 1790617829970140 us; error 29 us; skew 500 ppm
I20260928 17:50:29.971309 23822 webserver.cc:460] Webserver started at http://127.3.222.2:43805/ using document root <none> and password file <none>
I20260928 17:50:29.971539 23822 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:29.971585 23822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:29.971707 23822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:29.972697 23822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/data/instance:
uuid: "d0985e85d44444b0bc4621be12fdf742"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "e27502148251dd116c43b95c718df5bf"
server_key_iv: "292c5d68336b25923dd62dcd3fbaa058"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.973006 23822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance:
uuid: "d0985e85d44444b0bc4621be12fdf742"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "e27502148251dd116c43b95c718df5bf"
server_key_iv: "292c5d68336b25923dd62dcd3fbaa058"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.974143 23822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:29.974750 23836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.974893 23822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:29.974962 23822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/wal
uuid: "d0985e85d44444b0bc4621be12fdf742"
format_stamp: "Formatted at 2026-09-28 17:50:29 on dist-test-slave-w3bm"
server_key: "e27502148251dd116c43b95c718df5bf"
server_key_iv: "292c5d68336b25923dd62dcd3fbaa058"
server_key_version: "encryptionkey@0"
I20260928 17:50:29.975026 23822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:29.995057 23822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:29.995352 23822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:29.995477 23822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:29.995687 23822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:29.996042 23822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:29.996075 23822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:29.996104 23822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:29.996129 23822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:30.001421 23822 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.2:33159
I20260928 17:50:30.001688 23949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.2:33159 every 8 connection(s)
I20260928 17:50:30.001838 23822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/data/info.pb
I20260928 17:50:30.007201 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.002708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.2:51865 (local address 127.3.222.62:42329)
0928 17:50:30.002755 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:30.002757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:30.003022 (+ 265us) server_negotiation.cc:407] Connection header received
0928 17:50:30.003707 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:30.003710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:30.003723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:30.003786 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:30.004604 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.005122 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.006039 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.006231 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.006287 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.007012 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:30.007013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:30.007013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:30.007017 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:30.007030 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:30.007033 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:30.007072 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:30.007123 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 17:50:30.007144 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:50:30.007429 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.002677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:42329 (local address 127.3.222.2:51865)
0928 17:50:30.002894 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:30.002941 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:30.003596 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:30.003857 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:30.003861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:30.003871 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:30.004491 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.004498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.005207 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.005207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.005969 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.005971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.006295 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.006295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.006396 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.006892 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:50:30.006893 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:30.006921 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:50:30.006943 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:30.007074 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:30.007076 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:30.007078 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:50:30.007118 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 17:50:30.007193 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260928 17:50:30.007844 23950 heartbeater.cc:347] Connected to a master server at 127.3.222.62:42329
I20260928 17:50:30.007931 23950 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:30.008119 23950 heartbeater.cc:511] Master 127.3.222.62:42329 requested a full tablet report, sending...
I20260928 17:50:30.008140 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 23822
I20260928 17:50:30.008224 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/wal/instance
I20260928 17:50:30.008407 3960 external_mini_cluster.cc:1428] Setting key c85f283ea87bf73b466993765ba7df95
I20260928 17:50:30.008459 23630 ts_manager.cc:194] Registered new tserver with Master: d0985e85d44444b0bc4621be12fdf742 (127.3.222.2:33159)
I20260928 17:50:30.008874 23630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.2:51865
I20260928 17:50:30.009198 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.222.3:0
--local_ip_for_outbound_sockets=127.3.222.3
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:42329
--builtin_ntp_servers=127.3.222.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:50:30.088116 23953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:30.088303 23953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:30.088323 23953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:30.089452 23953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:30.089489 23953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:30.089514 23953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.222.3
I20260928 17:50:30.090924 23953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.222.3
--webserver_port=0
--tserver_master_addrs=127.3.222.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.222.3
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:30.091132 23953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:30.091429 23953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:30.096009 23961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:30.096022 23958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:30.096083 23959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:30.096277 23953 server_base.cc:1034] running on GCE node
I20260928 17:50:30.096480 23953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:30.096686 23953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:30.097854 23953 hybrid_clock.cc:648] HybridClock initialized: now 1790617830097834 us; error 37 us; skew 500 ppm
I20260928 17:50:30.098996 23953 webserver.cc:460] Webserver started at http://127.3.222.3:42995/ using document root <none> and password file <none>
I20260928 17:50:30.099201 23953 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:30.099285 23953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:30.099423 23953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:30.100420 23953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/data/instance:
uuid: "6a2b7294b3f4423a92d992ba8d5d043b"
format_stamp: "Formatted at 2026-09-28 17:50:30 on dist-test-slave-w3bm"
server_key: "6d1085b8bc6896f61679063a680d377f"
server_key_iv: "e80b3db14f03f49fbf78897eb2e0d64f"
server_key_version: "encryptionkey@0"
I20260928 17:50:30.100725 23953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance:
uuid: "6a2b7294b3f4423a92d992ba8d5d043b"
format_stamp: "Formatted at 2026-09-28 17:50:30 on dist-test-slave-w3bm"
server_key: "6d1085b8bc6896f61679063a680d377f"
server_key_iv: "e80b3db14f03f49fbf78897eb2e0d64f"
server_key_version: "encryptionkey@0"
I20260928 17:50:30.101902 23953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:50:30.102614 23967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:30.102787 23953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:50:30.102870 23953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/wal
uuid: "6a2b7294b3f4423a92d992ba8d5d043b"
format_stamp: "Formatted at 2026-09-28 17:50:30 on dist-test-slave-w3bm"
server_key: "6d1085b8bc6896f61679063a680d377f"
server_key_iv: "e80b3db14f03f49fbf78897eb2e0d64f"
server_key_version: "encryptionkey@0"
I20260928 17:50:30.102936 23953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:30.116123 23953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:30.116370 23953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:30.116493 23953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:30.116684 23953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:50:30.116966 23953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:50:30.116999 23953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:30.117025 23953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:50:30.117048 23953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:30.122221 23953 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.3:33349
I20260928 17:50:30.122280 24080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.3:33349 every 8 connection(s)
I20260928 17:50:30.122555 23953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/data/info.pb
I20260928 17:50:30.123425 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 23953
I20260928 17:50:30.123529 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/wal/instance
I20260928 17:50:30.123692 3960 external_mini_cluster.cc:1428] Setting key 473aaf929642bcdc3c532c1042271d55
I20260928 17:50:30.127597 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.123241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.3:51087 (local address 127.3.222.62:42329)
0928 17:50:30.123285 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:30.123287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:30.123507 (+ 220us) server_negotiation.cc:407] Connection header received
0928 17:50:30.124088 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:30.124089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:30.124101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:30.124150 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:30.124782 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.125257 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.126203 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.126434 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.126484 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.127401 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:30.127403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:30.127403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:30.127407 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:30.127422 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:30.127425 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:30.127462 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:30.127517 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 17:50:30.127540 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:50:30.127782 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.123202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:42329 (local address 127.3.222.3:51087)
0928 17:50:30.123397 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:30.123428 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:30.124002 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:30.124216 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:30.124220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:30.124230 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:30.124700 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.124704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.125334 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.125335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.126122 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.126125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.126508 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.126509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.126671 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.127187 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:50:30.127188 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:30.127245 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:50:30.127313 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:30.127468 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:30.127471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:30.127473 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:50:30.127520 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 17:50:30.127600 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260928 17:50:30.128235 24081 heartbeater.cc:347] Connected to a master server at 127.3.222.62:42329
I20260928 17:50:30.128311 24081 heartbeater.cc:464] Registering TS with master...
I20260928 17:50:30.128458 24081 heartbeater.cc:511] Master 127.3.222.62:42329 requested a full tablet report, sending...
I20260928 17:50:30.128762 23630 ts_manager.cc:194] Registered new tserver with Master: 6a2b7294b3f4423a92d992ba8d5d043b (127.3.222.3:33349)
I20260928 17:50:30.129119 23630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.222.3:51087
I20260928 17:50:30.135120 3960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:50:30.138827 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.136414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.3.222.62:42329)
0928 17:50:30.136489 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:30.136491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:30.136562 (+ 71us) server_negotiation.cc:407] Connection header received
0928 17:50:30.136609 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:30.136610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:30.136618 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:30.136659 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:30.136775 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.137236 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.137905 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.138087 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.138138 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.138664 (+ 526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:30.138665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:30.138665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:30.138668 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:30.138678 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:30.138680 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:30.138713 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:30.138750 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 17:50:30.138765 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 17:50:30.140395 23630 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:50:30.141979 23630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:50:30.147518 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.144041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:33159 (local address 127.0.0.1:59526)
0928 17:50:30.144189 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:30.144226 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:30.144297 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:30.144478 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:30.144482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:30.144485 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:30.144545 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.144547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.145585 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.145586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.146419 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.146421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.147224 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.147226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.147313 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.147328 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 17:50:30.147396 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:50:30.147453 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260928 17:50:30.147529 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.144117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.3.222.2:33159)
0928 17:50:30.144310 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:30.144315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:30.144328 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:50:30.144368 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:30.144370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:30.144382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:30.144454 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:30.144611 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.145527 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.146494 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.147133 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.147192 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.147300 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:30.147412 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 17:50:30.147446 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260928 17:50:30.148339 23884 tablet_service.cc:1467] Processing CreateTablet for tablet 775ebd58a6654fabbd65e6901229086d (DEFAULT_TABLE table=test-table [id=4853593e1ac04e3da776b3ede0582bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:30.148597 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.144041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:33349 (local address 127.0.0.1:60774)
0928 17:50:30.144399 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:30.144408 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:30.144446 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:30.145008 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:30.145009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:30.145010 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:30.145052 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.145054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.146293 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.146293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.147480 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.147483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.148367 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.148368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.148430 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.148433 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:50:30.148496 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 17:50:30.148532 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":52,"thread_start_us":48,"threads_started":1}
I20260928 17:50:30.148626 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.144355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.3.222.3:33349)
0928 17:50:30.144857 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:30.144861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:30.144869 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:50:30.144905 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:30.144907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:30.144921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:30.144988 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:30.145126 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.146213 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.147577 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.148283 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.148365 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.148435 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:30.148507 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 17:50:30.148545 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":87,"threads_started":1}
I20260928 17:50:30.148681 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.144216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:32895 (local address 127.0.0.1:49294)
0928 17:50:30.144693 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:30.144701 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:30.144753 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:30.145175 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:30.145176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:30.145178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:30.145216 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.145218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.146359 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.146360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.147510 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.147511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.148467 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.148468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.148529 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.148533 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:30.148579 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:50:30.148619 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":55,"threads_started":1}
I20260928 17:50:30.148701 23884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 775ebd58a6654fabbd65e6901229086d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:30.148902 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.144533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.3.222.1:32895)
0928 17:50:30.144688 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:30.144692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:30.144746 (+ 54us) server_negotiation.cc:407] Connection header received
0928 17:50:30.145077 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:30.145080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:30.145095 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:30.145169 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:30.145273 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.146298 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.147586 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.148283 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.148349 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.148431 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:30.148773 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 17:50:30.148813 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260928 17:50:30.149248 24015 tablet_service.cc:1467] Processing CreateTablet for tablet 775ebd58a6654fabbd65e6901229086d (DEFAULT_TABLE table=test-table [id=4853593e1ac04e3da776b3ede0582bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:30.149425 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 775ebd58a6654fabbd65e6901229086d (DEFAULT_TABLE table=test-table [id=4853593e1ac04e3da776b3ede0582bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:50:30.149555 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 775ebd58a6654fabbd65e6901229086d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:30.149654 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 775ebd58a6654fabbd65e6901229086d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:30.152194 24095 tablet_bootstrap.cc:492] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: Bootstrap starting.
I20260928 17:50:30.152279 24096 tablet_bootstrap.cc:492] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e: Bootstrap starting.
I20260928 17:50:30.152563 24097 tablet_bootstrap.cc:492] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: Bootstrap starting.
I20260928 17:50:30.152804 24095 tablet_bootstrap.cc:654] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:30.153017 24096 tablet_bootstrap.cc:654] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:30.153071 24095 log.cc:824] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:30.153215 24097 tablet_bootstrap.cc:654] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:30.153306 24096 log.cc:824] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:30.153491 24097 log.cc:824] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:30.153759 24095 tablet_bootstrap.cc:492] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: No bootstrap required, opened a new log
I20260928 17:50:30.153832 24095 ts_tablet_manager.cc:1397] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:30.154018 24096 tablet_bootstrap.cc:492] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e: No bootstrap required, opened a new log
I20260928 17:50:30.154078 24096 ts_tablet_manager.cc:1397] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:30.154295 24097 tablet_bootstrap.cc:492] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: No bootstrap required, opened a new log
I20260928 17:50:30.154367 24097 ts_tablet_manager.cc:1397] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:50:30.155718 24095 raft_consensus.cc:348] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } }
I20260928 17:50:30.155844 24095 raft_consensus.cc:374] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:30.155874 24095 raft_consensus.cc:729] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0985e85d44444b0bc4621be12fdf742, State: Initialized, Role: FOLLOWER
I20260928 17:50:30.156006 24095 consensus_queue.cc:260] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } }
I20260928 17:50:30.156230 24095 ts_tablet_manager.cc:1428] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:50:30.156347 23950 heartbeater.cc:503] Master 127.3.222.62:42329 was elected leader, sending a full tablet report...
I20260928 17:50:30.156445 24096 raft_consensus.cc:348] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } }
I20260928 17:50:30.156599 24096 raft_consensus.cc:374] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:30.156631 24096 raft_consensus.cc:729] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4ad15a615a4bd38781ded42503e45e, State: Initialized, Role: FOLLOWER
I20260928 17:50:30.156674 24097 raft_consensus.cc:348] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } }
I20260928 17:50:30.156751 24096 consensus_queue.cc:260] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } }
I20260928 17:50:30.156812 24097 raft_consensus.cc:374] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:30.156852 24097 raft_consensus.cc:729] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2b7294b3f4423a92d992ba8d5d043b, State: Initialized, Role: FOLLOWER
I20260928 17:50:30.156955 24097 consensus_queue.cc:260] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } }
I20260928 17:50:30.156991 24096 ts_tablet_manager.cc:1428] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:50:30.157104 23819 heartbeater.cc:503] Master 127.3.222.62:42329 was elected leader, sending a full tablet report...
I20260928 17:50:30.157181 24097 ts_tablet_manager.cc:1428] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 17:50:30.157327 24081 heartbeater.cc:503] Master 127.3.222.62:42329 was elected leader, sending a full tablet report...
I20260928 17:50:30.224501 24102 raft_consensus.cc:482] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:50:30.224624 24102 raft_consensus.cc:504] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } }
I20260928 17:50:30.224932 24102 leader_election.cc:288] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a2b7294b3f4423a92d992ba8d5d043b (127.3.222.3:33349), d0985e85d44444b0bc4621be12fdf742 (127.3.222.2:33159)
I20260928 17:50:30.227723 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.225113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.3:33349 (local address 127.3.222.1:57585)
0928 17:50:30.225175 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:30.225186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:30.225229 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:30.225372 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:30.225385 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:30.225387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:30.225456 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.225458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.226134 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.226135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.226932 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.226934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.227504 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.227505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.227575 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.227579 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:30.227627 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 17:50:30.227669 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 17:50:30.227736 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.225200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:57585 (local address 127.3.222.3:33349)
0928 17:50:30.225240 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:30.225242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:30.225249 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:50:30.225278 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:30.225279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:30.225288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:30.225340 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:30.225520 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.226063 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.227006 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.227439 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.227492 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.227501 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:30.227632 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 17:50:30.227668 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:50:30.227857 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.225113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.2:33159 (local address 127.3.222.1:44081)
0928 17:50:30.225379 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:30.225386 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:30.225431 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:30.225563 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:30.225563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:30.225565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:50:30.225629 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.225631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.226252 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.226252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.227052 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:30.227054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.227617 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.227618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:30.227698 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.227702 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:50:30.227754 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 17:50:30.227786 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":98,"thread_start_us":102,"threads_started":1}
I20260928 17:50:30.227866 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.225181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.222.1:44081 (local address 127.3.222.2:33159)
0928 17:50:30.225237 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:30.225239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:30.225461 (+ 222us) server_negotiation.cc:407] Connection header received
0928 17:50:30.225488 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:30.225489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:30.225501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:30.225549 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:50:30.225689 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.226182 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.227132 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.227541 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.227606 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.227615 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:30.227765 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 17:50:30.227802 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:50:30.228099 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "775ebd58a6654fabbd65e6901229086d" candidate_uuid: "7c4ad15a615a4bd38781ded42503e45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" is_pre_election: true
I20260928 17:50:30.228116 23904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "775ebd58a6654fabbd65e6901229086d" candidate_uuid: "7c4ad15a615a4bd38781ded42503e45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0985e85d44444b0bc4621be12fdf742" is_pre_election: true
I20260928 17:50:30.228233 23904 raft_consensus.cc:2454] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c4ad15a615a4bd38781ded42503e45e in term 0.
I20260928 17:50:30.228233 24035 raft_consensus.cc:2454] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c4ad15a615a4bd38781ded42503e45e in term 0.
I20260928 17:50:30.228454 23708 leader_election.cc:302] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c4ad15a615a4bd38781ded42503e45e, d0985e85d44444b0bc4621be12fdf742; no voters:
I20260928 17:50:30.228606 24102 raft_consensus.cc:2781] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:50:30.228662 24102 raft_consensus.cc:482] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:50:30.228677 24102 raft_consensus.cc:3037] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:30.229321 24102 raft_consensus.cc:504] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } }
I20260928 17:50:30.229426 24102 leader_election.cc:288] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [CANDIDATE]: Term 1 election: Requested vote from peers 6a2b7294b3f4423a92d992ba8d5d043b (127.3.222.3:33349), d0985e85d44444b0bc4621be12fdf742 (127.3.222.2:33159)
I20260928 17:50:30.229547 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "775ebd58a6654fabbd65e6901229086d" candidate_uuid: "7c4ad15a615a4bd38781ded42503e45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2b7294b3f4423a92d992ba8d5d043b"
I20260928 17:50:30.229590 23904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "775ebd58a6654fabbd65e6901229086d" candidate_uuid: "7c4ad15a615a4bd38781ded42503e45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0985e85d44444b0bc4621be12fdf742"
I20260928 17:50:30.229624 24035 raft_consensus.cc:3037] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:30.229652 23904 raft_consensus.cc:3037] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:30.230357 24035 raft_consensus.cc:2454] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c4ad15a615a4bd38781ded42503e45e in term 1.
I20260928 17:50:30.230352 23904 raft_consensus.cc:2454] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c4ad15a615a4bd38781ded42503e45e in term 1.
I20260928 17:50:30.230517 23707 leader_election.cc:302] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2b7294b3f4423a92d992ba8d5d043b, 7c4ad15a615a4bd38781ded42503e45e; no voters:
I20260928 17:50:30.230640 24102 raft_consensus.cc:2781] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:30.230818 24102 raft_consensus.cc:686] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [term 1 LEADER]: Becoming Leader. State: Replica: 7c4ad15a615a4bd38781ded42503e45e, State: Running, Role: LEADER
I20260928 17:50:30.230931 24102 consensus_queue.cc:237] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } }
I20260928 17:50:30.231825 23630 catalog_manager.cc:5495] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c4ad15a615a4bd38781ded42503e45e (127.3.222.1). New cstate: current_term: 1 leader_uuid: "7c4ad15a615a4bd38781ded42503e45e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c4ad15a615a4bd38781ded42503e45e" member_type: VOTER last_known_addr { host: "127.3.222.1" port: 32895 } health_report { overall_health: HEALTHY } } }
I20260928 17:50:30.240720 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.238706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.3.222.1:32895)
0928 17:50:30.238755 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:30.238757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:30.238764 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:50:30.238815 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:30.238816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:30.238831 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:30.238885 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:30.238983 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.239472 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.240165 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:30.240334 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:30.240374 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:30.240474 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:30.240562 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:30.240615 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:30.240629 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 17:50:30.240643 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:50:30.242285 24035 raft_consensus.cc:1261] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [term 1 FOLLOWER]: Refusing update from remote peer 7c4ad15a615a4bd38781ded42503e45e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:30.242329 23904 raft_consensus.cc:1261] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [term 1 FOLLOWER]: Refusing update from remote peer 7c4ad15a615a4bd38781ded42503e45e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:30.242588 24105 consensus_queue.cc:1035] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2b7294b3f4423a92d992ba8d5d043b" member_type: VOTER last_known_addr { host: "127.3.222.3" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:30.242698 24102 consensus_queue.cc:1035] T 775ebd58a6654fabbd65e6901229086d P 7c4ad15a615a4bd38781ded42503e45e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0985e85d44444b0bc4621be12fdf742" member_type: VOTER last_known_addr { host: "127.3.222.2" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:30.244889 24108 mvcc.cc:205] Tried to move back new op lower bound from 7334370632670031872 to 7334370632626630656. Current Snapshot: MvccSnapshot[applied={T|T < 7334370632670031872}]
I20260928 17:50:30.246052 24109 mvcc.cc:205] Tried to move back new op lower bound from 7334370632670031872 to 7334370632626630656. Current Snapshot: MvccSnapshot[applied={T|T < 7334370632670031872}]
I20260928 17:50:30.247112 23630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:30.247196 23630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:50:30.247884 23630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26c8cd598e074af99ec751648c5420af: Sending DeleteTablet for 3 replicas of tablet 775ebd58a6654fabbd65e6901229086d
I20260928 17:50:30.248188 24015 tablet_service.cc:1514] Processing DeleteTablet for tablet 775ebd58a6654fabbd65e6901229086d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:30 UTC) from {username='slave'} at 127.0.0.1:60774
I20260928 17:50:30.248204 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 775ebd58a6654fabbd65e6901229086d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:30 UTC) from {username='slave'} at 127.0.0.1:49294
I20260928 17:50:30.248221 23884 tablet_service.cc:1514] Processing DeleteTablet for tablet 775ebd58a6654fabbd65e6901229086d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:50:30 UTC) from {username='slave'} at 127.0.0.1:59526
I20260928 17:50:30.248373 24117 tablet_replica.cc:330] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: stopping tablet replica
I20260928 17:50:30.248428 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 23691
I20260928 17:50:30.248456 24117 raft_consensus.cc:2229] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:30.248982 24117 raft_consensus.cc:2258] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:30.249158 24118 tablet_replica.cc:330] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: stopping tablet replica
I20260928 17:50:30.249239 24118 raft_consensus.cc:2229] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:50:30.249316 24118 raft_consensus.cc:2258] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:50:30.249534 24117 ts_tablet_manager.cc:1905] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:50:30.249835 24118 ts_tablet_manager.cc:1905] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:50:30.251475 24117 ts_tablet_manager.cc:1918] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:30.251494 24118 ts_tablet_manager.cc:1918] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:50:30.251546 24117 log.cc:1196] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-2/wal/wals/775ebd58a6654fabbd65e6901229086d
I20260928 17:50:30.251550 24118 log.cc:1196] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: Deleting WAL directory at /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617599653609-3960-0/minicluster-data/ts-1/wal/wals/775ebd58a6654fabbd65e6901229086d
I20260928 17:50:30.251780 24118 ts_tablet_manager.cc:1939] T 775ebd58a6654fabbd65e6901229086d P d0985e85d44444b0bc4621be12fdf742: Deleting consensus metadata
I20260928 17:50:30.251787 24117 ts_tablet_manager.cc:1939] T 775ebd58a6654fabbd65e6901229086d P 6a2b7294b3f4423a92d992ba8d5d043b: Deleting consensus metadata
I20260928 17:50:30.252130 23616 catalog_manager.cc:4841] TS 6a2b7294b3f4423a92d992ba8d5d043b (127.3.222.3:33349): tablet 775ebd58a6654fabbd65e6901229086d (table test-table [id=4853593e1ac04e3da776b3ede0582bcc]) successfully deleted
I20260928 17:50:30.252233 23617 catalog_manager.cc:4841] TS d0985e85d44444b0bc4621be12fdf742 (127.3.222.2:33159): tablet 775ebd58a6654fabbd65e6901229086d (table test-table [id=4853593e1ac04e3da776b3ede0582bcc]) successfully deleted
W20260928 17:50:30.253623 23617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.222.1:32895 (error 108)
I20260928 17:50:30.254022 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 23822
I20260928 17:50:30.254048 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:30.253920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.1:32895 (local address 127.0.0.1:49320)
0928 17:50:30.253979 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:30.254020 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.1:32895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 17:50:30.259230 23617 catalog_manager.cc:4584] TS 7c4ad15a615a4bd38781ded42503e45e (127.3.222.1:32895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 775ebd58a6654fabbd65e6901229086d: Network error: Client connection negotiation failed: client connection to 127.3.222.1:32895: connect: Connection refused (error 111)
I20260928 17:50:30.259310 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 23953
I20260928 17:50:30.264035 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 23600
2026-09-28T17:50:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (687 ms)
[----------] 6 tests from EncryptionPolicyTest (7747 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:30 dist-test-slave-w3bm krb5kdc[24123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:30 dist-test-slave-w3bm krb5kdc[24123](info): set up 2 sockets
Sep 28 17:50:30 dist-test-slave-w3bm krb5kdc[24123](info): commencing operation
krb5kdc: starting...
W20260928 17:50:31.210908 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:31 dist-test-slave-w3bm krb5kdc[24123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:31.322299 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:31.400262 24136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:31.400452 24136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:31.400472 24136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:31.401589 24136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:31.401629 24136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:31.401643 24136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:31.401664 24136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:31.401686 24136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:31.403112 24136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:31.403340 24136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:31.403545 24136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:31.407905 24144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:31.407920 24142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:31.407920 24141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:31.408113 24136 server_base.cc:1034] running on GCE node
I20260928 17:50:31.408313 24136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:31.408542 24136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:31.409679 24136 hybrid_clock.cc:648] HybridClock initialized: now 1790617831409665 us; error 27 us; skew 500 ppm
Sep 28 17:50:31 dist-test-slave-w3bm krb5kdc[24123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:31.411948 24136 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:31.412344 24136 webserver.cc:460] Webserver started at http://127.0.0.1:46413/ using document root <none> and password file <none>
I20260928 17:50:31.412528 24136 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:31.412571 24136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:31.412689 24136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:31.413656 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "652b28fcb88448dea518f3b6ae166518"
format_stamp: "Formatted at 2026-09-28 17:50:31 on dist-test-slave-w3bm"
server_key: "d5bd4cac0053e4650ca0ab32cf19d287"
server_key_iv: "b6794b30874503ae537941774a3582f7"
server_key_version: "encryptionkey@0"
I20260928 17:50:31.413981 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "652b28fcb88448dea518f3b6ae166518"
format_stamp: "Formatted at 2026-09-28 17:50:31 on dist-test-slave-w3bm"
server_key: "d5bd4cac0053e4650ca0ab32cf19d287"
server_key_iv: "b6794b30874503ae537941774a3582f7"
server_key_version: "encryptionkey@0"
I20260928 17:50:31.415130 24136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:31.415756 24151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:31.415905 24136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:31.415987 24136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "652b28fcb88448dea518f3b6ae166518"
format_stamp: "Formatted at 2026-09-28 17:50:31 on dist-test-slave-w3bm"
server_key: "d5bd4cac0053e4650ca0ab32cf19d287"
server_key_iv: "b6794b30874503ae537941774a3582f7"
server_key_version: "encryptionkey@0"
I20260928 17:50:31.416046 24136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:31.424757 24136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:31.426359 24136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:31.426477 24136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:31.430047 24136 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43213
I20260928 17:50:31.430074 24203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43213 every 8 connection(s)
I20260928 17:50:31.430375 24136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:31.431231 24204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:31.433362 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518: Bootstrap starting.
I20260928 17:50:31.433892 24204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:31.434136 24204 log.cc:824] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:31.434738 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518: No bootstrap required, opened a new log
I20260928 17:50:31.436370 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 24136
I20260928 17:50:31.436318 24204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652b28fcb88448dea518f3b6ae166518" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43213 } }
I20260928 17:50:31.436470 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:31.436471 24204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:31.436532 24204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652b28fcb88448dea518f3b6ae166518, State: Initialized, Role: FOLLOWER
I20260928 17:50:31.436655 3960 external_mini_cluster.cc:1428] Setting key ff9766862a79ce4f268a8118e533f8ad
I20260928 17:50:31.436671 24204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652b28fcb88448dea518f3b6ae166518" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43213 } }
I20260928 17:50:31.436745 24204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:31.436780 24204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:31.436826 24204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:31.437667 24204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652b28fcb88448dea518f3b6ae166518" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43213 } }
I20260928 17:50:31.437770 24204 leader_election.cc:302] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 652b28fcb88448dea518f3b6ae166518; no voters:
I20260928 17:50:31.437914 24204 leader_election.cc:288] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:31.437943 24209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:31.438081 24209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [term 1 LEADER]: Becoming Leader. State: Replica: 652b28fcb88448dea518f3b6ae166518, State: Running, Role: LEADER
I20260928 17:50:31.438148 24204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:31.438166 24209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652b28fcb88448dea518f3b6ae166518" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43213 } }
I20260928 17:50:31.438519 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 652b28fcb88448dea518f3b6ae166518. Latest consensus state: current_term: 1 leader_uuid: "652b28fcb88448dea518f3b6ae166518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652b28fcb88448dea518f3b6ae166518" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43213 } } }
I20260928 17:50:31.438601 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:31.439275 24211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "652b28fcb88448dea518f3b6ae166518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652b28fcb88448dea518f3b6ae166518" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43213 } } }
I20260928 17:50:31.439378 24211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518 [sys.catalog]: This master's current role is: LEADER
W20260928 17:50:31.439749 24225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:31.439805 24225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:31.439862 24216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:31.440016 24216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:31.441460 24216 catalog_manager.cc:1347] Generated new cluster ID: 6d7f4d8cbce0457da9f8d54251aba2fb
I20260928 17:50:31.441510 24216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:50:31 dist-test-slave-w3bm krb5kdc[24123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 17:50:31.448267 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:31.437141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42831 (local address 127.0.0.1:43213)
0928 17:50:31.437321 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:31.437331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:31.437359 (+ 28us) server_negotiation.cc:407] Connection header received
0928 17:50:31.437449 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:31.437454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:31.437492 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:31.437594 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:31.438377 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:31.439785 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:31.440802 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:31.441383 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:31.441479 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:31.444278 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:31.444282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:31.444283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:31.444294 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:31.446625 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:31.447055 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:31.447058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:31.447058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:31.447162 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:31.447401 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:31.447402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:31.447402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:31.447677 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:31.447812 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:31.447938 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 17:50:31.448044 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260928 17:50:31.449189 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:31.452519 24216 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:31.453153 24216 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:31.457502 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:31.450525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37045 (local address 127.0.0.1:43213)
0928 17:50:31.450552 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:31.450554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:31.450668 (+ 114us) server_negotiation.cc:407] Connection header received
0928 17:50:31.450744 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:31.450745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:31.450753 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:31.450777 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:31.451430 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:31.452119 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:31.453155 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:31.453422 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:31.453460 (+ 38us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:31.454796 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:31.454798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:31.454798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:31.454805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:31.456349 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:31.456733 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:31.456734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:31.456735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:31.456781 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:31.456967 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:31.456968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:31.456968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:31.457071 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:31.457151 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:31.457342 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 17:50:31.457399 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20260928 17:50:31.458513 24216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 652b28fcb88448dea518f3b6ae166518: Generated new TSK 0
I20260928 17:50:31.458670 24216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:31.463204 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 24136
W20260928 17:50:31.467065 24121 connection.cc:535] client connection to 127.0.0.1:43213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43213): Connection reset by peer (error 104)
2026-09-28T17:50:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:31 dist-test-slave-w3bm krb5kdc[24235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:31 dist-test-slave-w3bm krb5kdc[24235](info): set up 2 sockets
Sep 28 17:50:31 dist-test-slave-w3bm krb5kdc[24235](info): commencing operation
krb5kdc: starting...
W20260928 17:50:32.406863 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:32 dist-test-slave-w3bm krb5kdc[24235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:32.517357 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:32.594743 24248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:32.594949 24248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:32.594969 24248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:32.596125 24248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:32.596165 24248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:32.596177 24248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:32.596189 24248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:32.596199 24248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:32.597569 24248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:32.597745 24248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:32.597908 24248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:32.602571 24256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:32.602622 24248 server_base.cc:1034] running on GCE node
W20260928 17:50:32.602646 24253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:32.602800 24254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:32.603083 24248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:32.603380 24248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:32.604532 24248 hybrid_clock.cc:648] HybridClock initialized: now 1790617832604522 us; error 30 us; skew 500 ppm
Sep 28 17:50:32 dist-test-slave-w3bm krb5kdc[24235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:32.606818 24248 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:32.607188 24248 webserver.cc:460] Webserver started at http://127.0.0.1:34723/ using document root <none> and password file <none>
I20260928 17:50:32.607414 24248 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:32.607460 24248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:32.607573 24248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:32.608565 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "f9e880721cf14b14852c3fae83f5a30a"
format_stamp: "Formatted at 2026-09-28 17:50:32 on dist-test-slave-w3bm"
server_key: "afe0858e18385b0e4d650d4b89b6da76"
server_key_iv: "e84c5e73b6e6bda476bfc0cf4512c85a"
server_key_version: "encryptionkey@0"
I20260928 17:50:32.608871 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "f9e880721cf14b14852c3fae83f5a30a"
format_stamp: "Formatted at 2026-09-28 17:50:32 on dist-test-slave-w3bm"
server_key: "afe0858e18385b0e4d650d4b89b6da76"
server_key_iv: "e84c5e73b6e6bda476bfc0cf4512c85a"
server_key_version: "encryptionkey@0"
I20260928 17:50:32.610090 24248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:32.610726 24263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:32.610868 24248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:32.610950 24248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "f9e880721cf14b14852c3fae83f5a30a"
format_stamp: "Formatted at 2026-09-28 17:50:32 on dist-test-slave-w3bm"
server_key: "afe0858e18385b0e4d650d4b89b6da76"
server_key_iv: "e84c5e73b6e6bda476bfc0cf4512c85a"
server_key_version: "encryptionkey@0"
I20260928 17:50:32.611011 24248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:32.621413 24248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:32.623199 24248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:32.623337 24248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:32.626761 24248 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45721
I20260928 17:50:32.626797 24315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45721 every 8 connection(s)
I20260928 17:50:32.627100 24248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:32.628038 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:32.630136 24316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a: Bootstrap starting.
I20260928 17:50:32.630649 24316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:32.630882 24316 log.cc:824] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:32.631389 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 24248
I20260928 17:50:32.631513 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:32.631594 24316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a: No bootstrap required, opened a new log
I20260928 17:50:32.631702 3960 external_mini_cluster.cc:1428] Setting key 85caafa432127124674f2761a39cf05c
I20260928 17:50:32.633442 24316 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e880721cf14b14852c3fae83f5a30a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } }
I20260928 17:50:32.633579 24316 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:32.633612 24316 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e880721cf14b14852c3fae83f5a30a, State: Initialized, Role: FOLLOWER
I20260928 17:50:32.633728 24316 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e880721cf14b14852c3fae83f5a30a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } }
I20260928 17:50:32.633785 24316 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:32.633814 24316 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:32.633869 24316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:32.634380 24316 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e880721cf14b14852c3fae83f5a30a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } }
I20260928 17:50:32.634471 24316 leader_election.cc:302] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9e880721cf14b14852c3fae83f5a30a; no voters:
I20260928 17:50:32.634617 24316 leader_election.cc:288] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:32.634651 24321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:32.634756 24321 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [term 1 LEADER]: Becoming Leader. State: Replica: f9e880721cf14b14852c3fae83f5a30a, State: Running, Role: LEADER
I20260928 17:50:32.634832 24321 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e880721cf14b14852c3fae83f5a30a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } }
I20260928 17:50:32.634861 24316 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:32.635170 24322 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9e880721cf14b14852c3fae83f5a30a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e880721cf14b14852c3fae83f5a30a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } } }
I20260928 17:50:32.635285 24322 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:32.635167 24321 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e880721cf14b14852c3fae83f5a30a. Latest consensus state: current_term: 1 leader_uuid: "f9e880721cf14b14852c3fae83f5a30a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e880721cf14b14852c3fae83f5a30a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } } }
I20260928 17:50:32.635466 24321 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:32.635573 24327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:32.636031 24327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:32.637575 24327 catalog_manager.cc:1347] Generated new cluster ID: 271dc3a3dffa4524a8e793a1bca4d84e
I20260928 17:50:32.637624 24327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:32.637760 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:32.632196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47569 (local address 127.0.0.1:45721)
0928 17:50:32.632368 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:32.632376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:32.632399 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:50:32.632798 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:32.632804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:32.632848 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:32.632962 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:32.633199 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:32.634411 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:32.635433 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:32.636017 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:32.636097 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:32.637145 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:32.637149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:32.637149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:32.637302 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:32.637331 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:32.637341 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:32.637407 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:32.637502 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 17:50:32.637530 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260928 17:50:32.640071 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:32.645726 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:32.641320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44569 (local address 127.0.0.1:45721)
0928 17:50:32.641361 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:32.641362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:32.643359 (+ 1997us) server_negotiation.cc:407] Connection header received
0928 17:50:32.643414 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:32.643415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:32.643426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:32.643488 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:32.643598 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:32.644045 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:32.644706 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:32.644912 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:32.644963 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:32.645510 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:32.645512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:32.645512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:32.645516 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:32.645530 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:32.645534 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:32.645582 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:32.645646 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 17:50:32.645655 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:50:32.675717 24327 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:32.676257 24327 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:32.682222 24327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9e880721cf14b14852c3fae83f5a30a: Generated new TSK 0
I20260928 17:50:32.682408 24327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:32.686838 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 24248
W20260928 17:50:32.690819 24233 connection.cc:535] client connection to 127.0.0.1:45721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45721): Connection reset by peer (error 104)
2026-09-28T17:50:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:32 dist-test-slave-w3bm krb5kdc[24347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:32 dist-test-slave-w3bm krb5kdc[24347](info): set up 2 sockets
Sep 28 17:50:32 dist-test-slave-w3bm krb5kdc[24347](info): commencing operation
krb5kdc: starting...
W20260928 17:50:33.630884 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:33 dist-test-slave-w3bm krb5kdc[24347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:33.741379 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:33.818377 24360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:33.818578 24360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:33.818598 24360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:33.819734 24360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:33.819773 24360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:33.819787 24360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:33.819797 24360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:33.819808 24360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:33.821192 24360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:33.821395 24360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:33.821599 24360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:33.826148 24368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:33.826196 24365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:33.826215 24366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:33.826349 24360 server_base.cc:1034] running on GCE node
I20260928 17:50:33.826644 24360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:33.826903 24360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:33.828033 24360 hybrid_clock.cc:648] HybridClock initialized: now 1790617833828012 us; error 37 us; skew 500 ppm
Sep 28 17:50:33 dist-test-slave-w3bm krb5kdc[24347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:33.830263 24360 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:33.830687 24360 webserver.cc:460] Webserver started at http://127.0.0.1:40995/ using document root <none> and password file <none>
I20260928 17:50:33.830875 24360 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:33.830919 24360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:33.831032 24360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:33.832046 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "d39b17f76cd44a78a92a7d607d1d26d5"
format_stamp: "Formatted at 2026-09-28 17:50:33 on dist-test-slave-w3bm"
server_key: "1ad9a3a2bfae4b6ba8c5ebb0a141726a"
server_key_iv: "8dab2374a4027968a38c8126ef24dd16"
server_key_version: "encryptionkey@0"
I20260928 17:50:33.832367 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "d39b17f76cd44a78a92a7d607d1d26d5"
format_stamp: "Formatted at 2026-09-28 17:50:33 on dist-test-slave-w3bm"
server_key: "1ad9a3a2bfae4b6ba8c5ebb0a141726a"
server_key_iv: "8dab2374a4027968a38c8126ef24dd16"
server_key_version: "encryptionkey@0"
I20260928 17:50:33.833561 24360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:33.834190 24375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:33.834331 24360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:33.834416 24360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "d39b17f76cd44a78a92a7d607d1d26d5"
format_stamp: "Formatted at 2026-09-28 17:50:33 on dist-test-slave-w3bm"
server_key: "1ad9a3a2bfae4b6ba8c5ebb0a141726a"
server_key_iv: "8dab2374a4027968a38c8126ef24dd16"
server_key_version: "encryptionkey@0"
I20260928 17:50:33.834473 24360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:33.837630 24360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:33.839579 24360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:33.839699 24360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:33.843245 24360 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32913
I20260928 17:50:33.843283 24427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32913 every 8 connection(s)
I20260928 17:50:33.843561 24360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:33.844398 24428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:33.845132 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 24360
I20260928 17:50:33.845216 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:33.845377 3960 external_mini_cluster.cc:1428] Setting key 30f389889584614182efc19a8b6b5840
I20260928 17:50:33.847250 24428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5: Bootstrap starting.
I20260928 17:50:33.847631 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:33.845827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52059 (local address 127.0.0.1:32913)
0928 17:50:33.845993 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:33.846001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:33.846034 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:50:33.846445 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:33.846453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:33.846486 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:33.846641 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:33.847102 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:33.847106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:33.847107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:33.847111 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:33.847150 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:33.847163 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:33.847255 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:33.847403 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 17:50:33.847441 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260928 17:50:33.847880 24428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:33.848160 24428 log.cc:824] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:33.848536 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:33.848824 24428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5: No bootstrap required, opened a new log
I20260928 17:50:33.850682 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:33.849825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35987 (local address 127.0.0.1:32913)
0928 17:50:33.849867 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:33.849869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:33.849877 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:50:33.849955 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:33.849956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:33.849960 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:33.850006 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:33.850467 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:33.850468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:33.850469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:33.850472 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:33.850485 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:33.850488 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:33.850532 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:33.850578 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 17:50:33.850588 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:50:33.850600 24428 raft_consensus.cc:348] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39b17f76cd44a78a92a7d607d1d26d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260928 17:50:33.850791 24428 raft_consensus.cc:374] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:33.850831 24428 raft_consensus.cc:729] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39b17f76cd44a78a92a7d607d1d26d5, State: Initialized, Role: FOLLOWER
I20260928 17:50:33.850940 24428 consensus_queue.cc:260] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39b17f76cd44a78a92a7d607d1d26d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260928 17:50:33.851025 24428 raft_consensus.cc:388] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:33.851109 24428 raft_consensus.cc:482] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:33.851163 24428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:33.851663 24428 raft_consensus.cc:504] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39b17f76cd44a78a92a7d607d1d26d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260928 17:50:33.851745 24428 leader_election.cc:302] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d39b17f76cd44a78a92a7d607d1d26d5; no voters:
I20260928 17:50:33.851881 24428 leader_election.cc:288] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:33.851917 24438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:33.852049 24438 raft_consensus.cc:686] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [term 1 LEADER]: Becoming Leader. State: Replica: d39b17f76cd44a78a92a7d607d1d26d5, State: Running, Role: LEADER
I20260928 17:50:33.852101 24428 sys_catalog.cc:564] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:33.852165 24438 consensus_queue.cc:237] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39b17f76cd44a78a92a7d607d1d26d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260928 17:50:33.852494 24440 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d39b17f76cd44a78a92a7d607d1d26d5. Latest consensus state: current_term: 1 leader_uuid: "d39b17f76cd44a78a92a7d607d1d26d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39b17f76cd44a78a92a7d607d1d26d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } } }
I20260928 17:50:33.852485 24439 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d39b17f76cd44a78a92a7d607d1d26d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39b17f76cd44a78a92a7d607d1d26d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } } }
I20260928 17:50:33.852582 24440 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:33.852608 24439 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5 [sys.catalog]: This master's current role is: LEADER
W20260928 17:50:33.853653 24454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:33.853703 24454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:33.853772 24446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:33.853955 24446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:33.855172 24446 catalog_manager.cc:1347] Generated new cluster ID: 502a3137bf1340b0aac7cfc73717dbba
I20260928 17:50:33.855239 24446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:33.867547 24446 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:33.867710 24446 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:33.879515 24446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d39b17f76cd44a78a92a7d607d1d26d5: Generated new TSK 0
I20260928 17:50:33.879714 24446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:33.890230 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 24360
2026-09-28T17:50:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:33 dist-test-slave-w3bm krb5kdc[24459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:33 dist-test-slave-w3bm krb5kdc[24459](info): set up 2 sockets
Sep 28 17:50:33 dist-test-slave-w3bm krb5kdc[24459](info): commencing operation
krb5kdc: starting...
W20260928 17:50:34.826936 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:34 dist-test-slave-w3bm krb5kdc[24459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:34.943925 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:35.023813 24472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:35.024011 24472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:35.024041 24472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:35.025202 24472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:35.025241 24472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:35.025260 24472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:35.025275 24472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:35.025291 24472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:35.026719 24472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:35.026938 24472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:35.027130 24472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:35.031611 24478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:35.031659 24480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:35.031672 24477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:35.031765 24472 server_base.cc:1034] running on GCE node
I20260928 17:50:35.031970 24472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:35.032204 24472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:35.033356 24472 hybrid_clock.cc:648] HybridClock initialized: now 1790617835033341 us; error 27 us; skew 500 ppm
Sep 28 17:50:35 dist-test-slave-w3bm krb5kdc[24459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:35.035602 24472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:35.036013 24472 webserver.cc:460] Webserver started at http://127.0.0.1:33841/ using document root <none> and password file <none>
I20260928 17:50:35.036201 24472 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:35.036242 24472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:35.036352 24472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:35.037317 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "b9c18695b63e4428bbe6e5df2751ae1b"
format_stamp: "Formatted at 2026-09-28 17:50:35 on dist-test-slave-w3bm"
server_key: "c20f79199d82fb6272cd95f95a8b3908"
server_key_iv: "eed0664f8114bcfb4c77eccf1028c964"
server_key_version: "encryptionkey@0"
I20260928 17:50:35.037629 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "b9c18695b63e4428bbe6e5df2751ae1b"
format_stamp: "Formatted at 2026-09-28 17:50:35 on dist-test-slave-w3bm"
server_key: "c20f79199d82fb6272cd95f95a8b3908"
server_key_iv: "eed0664f8114bcfb4c77eccf1028c964"
server_key_version: "encryptionkey@0"
I20260928 17:50:35.038810 24472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:35.039573 24487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:35.039718 24472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:50:35.039798 24472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "b9c18695b63e4428bbe6e5df2751ae1b"
format_stamp: "Formatted at 2026-09-28 17:50:35 on dist-test-slave-w3bm"
server_key: "c20f79199d82fb6272cd95f95a8b3908"
server_key_iv: "eed0664f8114bcfb4c77eccf1028c964"
server_key_version: "encryptionkey@0"
I20260928 17:50:35.039855 24472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:35.049225 24472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:35.050941 24472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:35.051062 24472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:35.054646 24472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46045
I20260928 17:50:35.054675 24539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46045 every 8 connection(s)
I20260928 17:50:35.054971 24472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:35.055857 24540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:35.057919 24540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b: Bootstrap starting.
I20260928 17:50:35.058110 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 24472
I20260928 17:50:35.058215 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:35.058365 3960 external_mini_cluster.cc:1428] Setting key e8255333b7a8d14858e7bfd370a11322
I20260928 17:50:35.058519 24540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:35.058817 24540 log.cc:824] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:35.059495 24540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b: No bootstrap required, opened a new log
I20260928 17:50:35.061550 24540 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c18695b63e4428bbe6e5df2751ae1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260928 17:50:35.061718 24540 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:35.061750 24540 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c18695b63e4428bbe6e5df2751ae1b, State: Initialized, Role: FOLLOWER
I20260928 17:50:35.061868 24540 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c18695b63e4428bbe6e5df2751ae1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260928 17:50:35.061932 24540 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:35.061959 24540 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:35.062000 24540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:35.062530 24540 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c18695b63e4428bbe6e5df2751ae1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260928 17:50:35.062615 24540 leader_election.cc:302] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9c18695b63e4428bbe6e5df2751ae1b; no voters:
I20260928 17:50:35.062763 24540 leader_election.cc:288] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:35.062844 24545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:35.063024 24545 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [term 1 LEADER]: Becoming Leader. State: Replica: b9c18695b63e4428bbe6e5df2751ae1b, State: Running, Role: LEADER
I20260928 17:50:35.063022 24540 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:35.063129 24545 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c18695b63e4428bbe6e5df2751ae1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260928 17:50:35.063622 24547 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9c18695b63e4428bbe6e5df2751ae1b. Latest consensus state: current_term: 1 leader_uuid: "b9c18695b63e4428bbe6e5df2751ae1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c18695b63e4428bbe6e5df2751ae1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260928 17:50:35.063717 24547 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:35.063637 24546 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9c18695b63e4428bbe6e5df2751ae1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c18695b63e4428bbe6e5df2751ae1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260928 17:50:35.063791 24546 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:35.064564 24554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 17:50:35 dist-test-slave-w3bm krb5kdc[24459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 17:50:35.065106 24554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:35.066543 24554 catalog_manager.cc:1347] Generated new cluster ID: db44884fd5f849b2ac22dd2f1b63fc9b
I20260928 17:50:35.066603 24554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:35.068552 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:35.058808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38267 (local address 127.0.0.1:46045)
0928 17:50:35.058983 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:35.058991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:35.059021 (+ 30us) server_negotiation.cc:407] Connection header received
0928 17:50:35.059518 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:35.059525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:35.059573 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:35.059708 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:35.060404 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:35.061244 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:35.062239 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:35.062670 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:35.062769 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:35.065431 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:35.065435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:35.065435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:35.065447 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:35.067103 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:35.067493 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:35.067495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:35.067496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:35.067584 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:35.067753 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:35.067754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:35.067754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:35.067984 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:35.068121 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:35.068222 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 17:50:35.068351 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260928 17:50:35.069370 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:35.080652 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:35.070586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53925 (local address 127.0.0.1:46045)
0928 17:50:35.070618 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:35.070620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:35.075435 (+ 4815us) server_negotiation.cc:407] Connection header received
0928 17:50:35.075506 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:35.075507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:35.075517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:35.075562 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:35.076052 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:35.076537 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:35.077205 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:35.077416 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:35.077469 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:35.078501 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:35.078502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:35.078503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:35.078509 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:35.079701 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:35.079993 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:35.079995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:35.079995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:35.080046 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:35.080207 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:35.080208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:35.080208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:35.080310 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:35.080391 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:35.080494 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 17:50:35.080562 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:50:35.095204 24554 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:35.095753 24554 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:35.104136 24554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9c18695b63e4428bbe6e5df2751ae1b: Generated new TSK 0
I20260928 17:50:35.104327 24554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:35.115820 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 24472
2026-09-28T17:50:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:35 dist-test-slave-w3bm krb5kdc[24571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:35 dist-test-slave-w3bm krb5kdc[24571](info): set up 2 sockets
Sep 28 17:50:35 dist-test-slave-w3bm krb5kdc[24571](info): commencing operation
krb5kdc: starting...
W20260928 17:50:36.062922 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:36 dist-test-slave-w3bm krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:36.174067 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:36.251351 24584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:36.251547 24584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:36.251576 24584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:36.252691 24584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:36.252729 24584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:36.252748 24584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:36.252763 24584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:36.252779 24584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:36.254204 24584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:36.254431 24584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:36.254619 24584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:36.259935 24592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:36.260085 24584 server_base.cc:1034] running on GCE node
W20260928 17:50:36.260087 24590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:36.260174 24589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:36.260454 24584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:36.260805 24584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:36.261968 24584 hybrid_clock.cc:648] HybridClock initialized: now 1790617836261945 us; error 38 us; skew 500 ppm
Sep 28 17:50:36 dist-test-slave-w3bm krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:36.265228 24584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:36.265589 24584 webserver.cc:460] Webserver started at http://127.0.0.1:36845/ using document root <none> and password file <none>
I20260928 17:50:36.265781 24584 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:36.265820 24584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:36.265903 24584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:36.266927 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "73972a513cc94f70821b90f5c3a53f18"
format_stamp: "Formatted at 2026-09-28 17:50:36 on dist-test-slave-w3bm"
server_key: "1eff7e93321ab6065186e8799f0418d6"
server_key_iv: "685950c118bd2ba3a1a65e9f3322950a"
server_key_version: "encryptionkey@0"
I20260928 17:50:36.267228 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "73972a513cc94f70821b90f5c3a53f18"
format_stamp: "Formatted at 2026-09-28 17:50:36 on dist-test-slave-w3bm"
server_key: "1eff7e93321ab6065186e8799f0418d6"
server_key_iv: "685950c118bd2ba3a1a65e9f3322950a"
server_key_version: "encryptionkey@0"
I20260928 17:50:36.268370 24584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:36.269038 24599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:36.269171 24584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:36.269250 24584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "73972a513cc94f70821b90f5c3a53f18"
format_stamp: "Formatted at 2026-09-28 17:50:36 on dist-test-slave-w3bm"
server_key: "1eff7e93321ab6065186e8799f0418d6"
server_key_iv: "685950c118bd2ba3a1a65e9f3322950a"
server_key_version: "encryptionkey@0"
I20260928 17:50:36.269315 24584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:36.291061 24584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:36.293056 24584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:36.293176 24584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:36.296862 24584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42535
I20260928 17:50:36.296908 24651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42535 every 8 connection(s)
I20260928 17:50:36.297215 24584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:36.298121 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:36.298319 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 24584
I20260928 17:50:36.298424 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:36.298602 3960 external_mini_cluster.cc:1428] Setting key 34d554b918309c2c7bacc253b52e32fc
I20260928 17:50:36.301082 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18: Bootstrap starting.
I20260928 17:50:36.301750 24652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:36.302042 24652 log.cc:824] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:36.302711 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18: No bootstrap required, opened a new log
I20260928 17:50:36.303700 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:36.299032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53837 (local address 127.0.0.1:42535)
0928 17:50:36.299177 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:36.299184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:36.299230 (+ 46us) server_negotiation.cc:407] Connection header received
0928 17:50:36.299657 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:36.299663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:36.299708 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:36.299816 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:36.300086 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:36.300935 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:36.301949 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:36.302355 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:36.302444 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:36.303023 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:36.303027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:36.303027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:36.303238 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:36.303276 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:36.303291 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:36.303359 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:36.303414 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 17:50:36.303478 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260928 17:50:36.304459 24652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73972a513cc94f70821b90f5c3a53f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260928 17:50:36.304615 24652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:36.304649 24652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73972a513cc94f70821b90f5c3a53f18, State: Initialized, Role: FOLLOWER
I20260928 17:50:36.304755 24652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73972a513cc94f70821b90f5c3a53f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260928 17:50:36.304838 24652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:36.304876 24652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:36.304924 24652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:36.304994 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:36.305549 24652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73972a513cc94f70821b90f5c3a53f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260928 17:50:36.305655 24652 leader_election.cc:302] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73972a513cc94f70821b90f5c3a53f18; no voters:
I20260928 17:50:36.305804 24652 leader_election.cc:288] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:36.305851 24657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:36.306033 24657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [term 1 LEADER]: Becoming Leader. State: Replica: 73972a513cc94f70821b90f5c3a53f18, State: Running, Role: LEADER
I20260928 17:50:36.306041 24652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:36.306198 24657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73972a513cc94f70821b90f5c3a53f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260928 17:50:36.306488 24657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73972a513cc94f70821b90f5c3a53f18. Latest consensus state: current_term: 1 leader_uuid: "73972a513cc94f70821b90f5c3a53f18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73972a513cc94f70821b90f5c3a53f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } } }
I20260928 17:50:36.306559 24657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:36.306521 24658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73972a513cc94f70821b90f5c3a53f18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73972a513cc94f70821b90f5c3a53f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } } }
I20260928 17:50:36.306653 24658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:36.307282 24669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:36.307763 24669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:36.309196 24669 catalog_manager.cc:1347] Generated new cluster ID: 8346c79fe3f141a38ccbef2e3e23a55a
I20260928 17:50:36.309237 24669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:36.310827 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:36.307547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34479 (local address 127.0.0.1:42535)
0928 17:50:36.307583 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:36.307585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:36.307799 (+ 214us) server_negotiation.cc:407] Connection header received
0928 17:50:36.307845 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:36.307846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:36.307858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:36.307885 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:36.308048 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:36.308713 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:36.309660 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:36.309952 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:36.310019 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:36.310608 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:36.310609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:36.310610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:36.310614 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:36.310628 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:36.310632 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:36.310674 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:36.310740 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 17:50:36.310760 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:50:36.332561 24669 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:36.333225 24669 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:36.337023 24669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73972a513cc94f70821b90f5c3a53f18: Generated new TSK 0
I20260928 17:50:36.337194 24669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:36.342327 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 24584
2026-09-28T17:50:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:36 dist-test-slave-w3bm krb5kdc[24683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:36 dist-test-slave-w3bm krb5kdc[24683](info): set up 2 sockets
Sep 28 17:50:36 dist-test-slave-w3bm krb5kdc[24683](info): commencing operation
krb5kdc: starting...
W20260928 17:50:37.282935 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:37 dist-test-slave-w3bm krb5kdc[24683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:37.393842 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:37.470952 24696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:37.471189 24696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:37.471238 24696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:37.472404 24696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:37.472445 24696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:37.472463 24696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:37.472481 24696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:37.472496 24696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:37.473955 24696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:37.474218 24696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:37.474428 24696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:37.479000 24702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:37.479024 24704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:37.479152 24701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:37.479097 24696 server_base.cc:1034] running on GCE node
I20260928 17:50:37.479463 24696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:37.479758 24696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:37.480923 24696 hybrid_clock.cc:648] HybridClock initialized: now 1790617837480908 us; error 33 us; skew 500 ppm
Sep 28 17:50:37 dist-test-slave-w3bm krb5kdc[24683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:37.483378 24696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:37.483965 24696 webserver.cc:460] Webserver started at http://127.0.0.1:44393/ using document root <none> and password file <none>
I20260928 17:50:37.484210 24696 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:37.484258 24696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:37.484385 24696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:37.485417 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "1e6860f40df8426eab5bfb4281ae5fe0"
format_stamp: "Formatted at 2026-09-28 17:50:37 on dist-test-slave-w3bm"
server_key: "02d59e5ba5622fc135f066d1d0609fbc"
server_key_iv: "a9bf39dbc28131f41f33c1b757118488"
server_key_version: "encryptionkey@0"
I20260928 17:50:37.485744 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "1e6860f40df8426eab5bfb4281ae5fe0"
format_stamp: "Formatted at 2026-09-28 17:50:37 on dist-test-slave-w3bm"
server_key: "02d59e5ba5622fc135f066d1d0609fbc"
server_key_iv: "a9bf39dbc28131f41f33c1b757118488"
server_key_version: "encryptionkey@0"
I20260928 17:50:37.486961 24696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:37.487711 24711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:37.487856 24696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:37.487936 24696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "1e6860f40df8426eab5bfb4281ae5fe0"
format_stamp: "Formatted at 2026-09-28 17:50:37 on dist-test-slave-w3bm"
server_key: "02d59e5ba5622fc135f066d1d0609fbc"
server_key_iv: "a9bf39dbc28131f41f33c1b757118488"
server_key_version: "encryptionkey@0"
I20260928 17:50:37.488004 24696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:37.491106 24696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:37.492857 24696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:37.493028 24696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:37.496682 24696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45033
I20260928 17:50:37.496755 24763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45033 every 8 connection(s)
I20260928 17:50:37.497044 24696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:37.497795 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 24696
I20260928 17:50:37.497893 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:37.497977 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:37.498095 3960 external_mini_cluster.cc:1428] Setting key 28ffb4718f4805eb1fda4cfbfa4ab596
I20260928 17:50:37.500602 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:37.498566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51985 (local address 127.0.0.1:45033)
0928 17:50:37.498757 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:37.498767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:37.498801 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:50:37.499364 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:37.499373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:37.499407 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:37.499565 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:37.500208 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:37.500212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:37.500213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:37.500216 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:37.500262 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:37.500277 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:37.500348 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:37.500385 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 17:50:37.500423 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260928 17:50:37.501037 24764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0: Bootstrap starting.
I20260928 17:50:37.501585 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:37.501701 24764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:37.501968 24764 log.cc:824] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:37.502693 24764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0: No bootstrap required, opened a new log
I20260928 17:50:37.504189 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:37.503111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42367 (local address 127.0.0.1:45033)
0928 17:50:37.503146 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:37.503147 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:37.503201 (+ 54us) server_negotiation.cc:407] Connection header received
0928 17:50:37.503292 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:37.503293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:37.503297 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:37.503358 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:37.503982 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:37.503983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:37.503984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:37.503987 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:37.504003 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:37.504007 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:37.504067 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:37.504101 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 17:50:37.504112 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:50:37.505174 24764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6860f40df8426eab5bfb4281ae5fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260928 17:50:37.505316 24764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:37.505337 24764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6860f40df8426eab5bfb4281ae5fe0, State: Initialized, Role: FOLLOWER
I20260928 17:50:37.505437 24764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6860f40df8426eab5bfb4281ae5fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260928 17:50:37.505537 24764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:37.505569 24764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:37.505609 24764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:37.506145 24764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6860f40df8426eab5bfb4281ae5fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260928 17:50:37.506228 24764 leader_election.cc:302] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e6860f40df8426eab5bfb4281ae5fe0; no voters:
I20260928 17:50:37.506389 24764 leader_election.cc:288] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:37.506506 24774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:37.506649 24774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [term 1 LEADER]: Becoming Leader. State: Replica: 1e6860f40df8426eab5bfb4281ae5fe0, State: Running, Role: LEADER
I20260928 17:50:37.506707 24764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:37.506731 24774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6860f40df8426eab5bfb4281ae5fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260928 17:50:37.507045 24775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e6860f40df8426eab5bfb4281ae5fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6860f40df8426eab5bfb4281ae5fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } } }
I20260928 17:50:37.507061 24776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e6860f40df8426eab5bfb4281ae5fe0. Latest consensus state: current_term: 1 leader_uuid: "1e6860f40df8426eab5bfb4281ae5fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6860f40df8426eab5bfb4281ae5fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } } }
I20260928 17:50:37.507244 24775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:37.507265 24776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:37.507532 24782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:37.507963 24782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:37.509361 24782 catalog_manager.cc:1347] Generated new cluster ID: 86fdca541dae476a8e68cc54e5ec7d86
I20260928 17:50:37.509403 24782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:37.515588 24782 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:37.515726 24782 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:37.526587 24782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e6860f40df8426eab5bfb4281ae5fe0: Generated new TSK 0
I20260928 17:50:37.526785 24782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:37.530769 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 24696
2026-09-28T17:50:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:37 dist-test-slave-w3bm krb5kdc[24795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:37 dist-test-slave-w3bm krb5kdc[24795](info): set up 2 sockets
Sep 28 17:50:37 dist-test-slave-w3bm krb5kdc[24795](info): commencing operation
krb5kdc: starting...
W20260928 17:50:38.494956 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:38 dist-test-slave-w3bm krb5kdc[24795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:38.606022 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:38.682921 24808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:38.683120 24808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:38.683149 24808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:38.684278 24808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:38.684320 24808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:38.684340 24808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:38.684355 24808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:38.684370 24808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:38.685791 24808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:38.686020 24808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:38.686213 24808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:38.690740 24814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:38.690864 24808 server_base.cc:1034] running on GCE node
W20260928 17:50:38.690730 24816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:38.690732 24813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:38.691188 24808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:38.691473 24808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:38.692631 24808 hybrid_clock.cc:648] HybridClock initialized: now 1790617838692617 us; error 28 us; skew 500 ppm
Sep 28 17:50:38 dist-test-slave-w3bm krb5kdc[24795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:38.699087 24808 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:38.699538 24808 webserver.cc:460] Webserver started at http://127.0.0.1:39647/ using document root <none> and password file <none>
I20260928 17:50:38.699730 24808 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:38.699781 24808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:38.699890 24808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:38.700937 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "df1220b3023549cd9dd96a21035dfec0"
format_stamp: "Formatted at 2026-09-28 17:50:38 on dist-test-slave-w3bm"
server_key: "3f5380f831d1534b1c881c9516237c9b"
server_key_iv: "6d5ebe35b6cc3a6004d92c510f80c231"
server_key_version: "encryptionkey@0"
I20260928 17:50:38.701256 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "df1220b3023549cd9dd96a21035dfec0"
format_stamp: "Formatted at 2026-09-28 17:50:38 on dist-test-slave-w3bm"
server_key: "3f5380f831d1534b1c881c9516237c9b"
server_key_iv: "6d5ebe35b6cc3a6004d92c510f80c231"
server_key_version: "encryptionkey@0"
I20260928 17:50:38.702476 24808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:38.703186 24823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:38.703365 24808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:38.703450 24808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "df1220b3023549cd9dd96a21035dfec0"
format_stamp: "Formatted at 2026-09-28 17:50:38 on dist-test-slave-w3bm"
server_key: "3f5380f831d1534b1c881c9516237c9b"
server_key_iv: "6d5ebe35b6cc3a6004d92c510f80c231"
server_key_version: "encryptionkey@0"
I20260928 17:50:38.703511 24808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:38.715979 24808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:38.718022 24808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:38.718161 24808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:38.722216 24808 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41409
I20260928 17:50:38.722251 24875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41409 every 8 connection(s)
I20260928 17:50:38.722573 24808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:38.723526 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:38.725677 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0: Bootstrap starting.
I20260928 17:50:38.726195 24876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:38.726430 24876 log.cc:824] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:38.727073 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0: No bootstrap required, opened a new log
I20260928 17:50:38.728600 24876 raft_consensus.cc:348] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1220b3023549cd9dd96a21035dfec0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260928 17:50:38.728732 24876 raft_consensus.cc:374] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:38.728753 24876 raft_consensus.cc:729] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1220b3023549cd9dd96a21035dfec0, State: Initialized, Role: FOLLOWER
I20260928 17:50:38.728874 24876 consensus_queue.cc:260] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1220b3023549cd9dd96a21035dfec0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260928 17:50:38.728932 24876 raft_consensus.cc:388] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:38.728951 24876 raft_consensus.cc:482] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:38.728978 24876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:38.729437 24876 raft_consensus.cc:504] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1220b3023549cd9dd96a21035dfec0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260928 17:50:38.729519 24876 leader_election.cc:302] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df1220b3023549cd9dd96a21035dfec0; no voters:
I20260928 17:50:38.729671 24876 leader_election.cc:288] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:38.729712 24879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:38.729851 24879 raft_consensus.cc:686] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [term 1 LEADER]: Becoming Leader. State: Replica: df1220b3023549cd9dd96a21035dfec0, State: Running, Role: LEADER
I20260928 17:50:38.729892 24876 sys_catalog.cc:564] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:38.729941 24879 consensus_queue.cc:237] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1220b3023549cd9dd96a21035dfec0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260928 17:50:38.730265 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 24808
I20260928 17:50:38.730386 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:38.730419 24880 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df1220b3023549cd9dd96a21035dfec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1220b3023549cd9dd96a21035dfec0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } } }
I20260928 17:50:38.730486 24880 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:38.730620 3960 external_mini_cluster.cc:1428] Setting key 1579aad21bfb796136a236bf3c0956b1
I20260928 17:50:38.730733 24881 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1220b3023549cd9dd96a21035dfec0. Latest consensus state: current_term: 1 leader_uuid: "df1220b3023549cd9dd96a21035dfec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1220b3023549cd9dd96a21035dfec0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } } }
I20260928 17:50:38.730816 24881 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:38.731060 24888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:38.731650 24888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:38.734617 24888 catalog_manager.cc:1347] Generated new cluster ID: 039f80100de949b6b060420250321818
I20260928 17:50:38.734673 24888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:50:38 dist-test-slave-w3bm krb5kdc[24795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 17:50:38.740932 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:38.731191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:55981 (local address 127.0.0.1:41409)
0928 17:50:38.731402 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:38.731411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:38.731441 (+ 30us) server_negotiation.cc:407] Connection header received
0928 17:50:38.731562 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:38.731570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:38.731630 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:38.732591 (+ 961us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:38.732683 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:38.734096 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:38.734949 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:38.735402 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:38.735500 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:38.737620 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:38.737624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:38.737624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:38.737635 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:38.739561 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:38.739911 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:38.739914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:38.739914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:38.740005 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:38.740201 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:38.740201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:38.740202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:38.740420 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:38.740567 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:38.740681 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 17:50:38.740777 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260928 17:50:38.741724 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:38.752890 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:38.742926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:53591 (local address 127.0.0.1:41409)
0928 17:50:38.742959 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:38.742960 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:38.747386 (+ 4426us) server_negotiation.cc:407] Connection header received
0928 17:50:38.747453 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:38.747454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:38.747465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:38.747513 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:38.748086 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:38.748567 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:38.749311 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:38.749528 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:38.749586 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:38.750647 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:38.750648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:38.750649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:38.750655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:38.751846 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:38.752170 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:38.752171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:38.752171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:38.752220 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:38.752402 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:38.752403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:38.752403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:38.752501 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:38.752587 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:38.752741 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 17:50:38.752798 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:50:38.753216 24888 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:38.753922 24888 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:38.764334 24888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df1220b3023549cd9dd96a21035dfec0: Generated new TSK 0
I20260928 17:50:38.764508 24888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:38.768105 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 24808
W20260928 17:50:38.771637 24793 connection.cc:535] client connection to 127.0.0.1:41409 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41409): Connection reset by peer (error 104)
2026-09-28T17:50:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:38 dist-test-slave-w3bm krb5kdc[24907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:38 dist-test-slave-w3bm krb5kdc[24907](info): set up 2 sockets
Sep 28 17:50:38 dist-test-slave-w3bm krb5kdc[24907](info): commencing operation
krb5kdc: starting...
W20260928 17:50:39.706982 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:39 dist-test-slave-w3bm krb5kdc[24907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:39.818295 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:39.896153 24920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:39.896358 24920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:39.896387 24920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:39.897513 24920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:39.897554 24920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:39.897580 24920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:39.897598 24920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:39.897612 24920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:39.899055 24920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:39.899317 24920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:39.899511 24920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:39.903999 24926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:39.904070 24920 server_base.cc:1034] running on GCE node
W20260928 17:50:39.904008 24925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:39.904008 24928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:39.904430 24920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:39.904711 24920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:39.905861 24920 hybrid_clock.cc:648] HybridClock initialized: now 1790617839905847 us; error 28 us; skew 500 ppm
Sep 28 17:50:39 dist-test-slave-w3bm krb5kdc[24907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:39.908249 24920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:39.908663 24920 webserver.cc:460] Webserver started at http://127.0.0.1:44857/ using document root <none> and password file <none>
I20260928 17:50:39.908846 24920 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:39.908885 24920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:39.908970 24920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:39.909952 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "49ed743946a74e6c87a607d5343e1401"
format_stamp: "Formatted at 2026-09-28 17:50:39 on dist-test-slave-w3bm"
server_key: "71981e71f1827444c7b48a5e69b9834e"
server_key_iv: "23f52c468d6c4672144e64c814ed013d"
server_key_version: "encryptionkey@0"
I20260928 17:50:39.910236 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "49ed743946a74e6c87a607d5343e1401"
format_stamp: "Formatted at 2026-09-28 17:50:39 on dist-test-slave-w3bm"
server_key: "71981e71f1827444c7b48a5e69b9834e"
server_key_iv: "23f52c468d6c4672144e64c814ed013d"
server_key_version: "encryptionkey@0"
I20260928 17:50:39.911441 24920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:39.912045 24935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:39.912196 24920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:39.912281 24920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "49ed743946a74e6c87a607d5343e1401"
format_stamp: "Formatted at 2026-09-28 17:50:39 on dist-test-slave-w3bm"
server_key: "71981e71f1827444c7b48a5e69b9834e"
server_key_iv: "23f52c468d6c4672144e64c814ed013d"
server_key_version: "encryptionkey@0"
I20260928 17:50:39.912343 24920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:39.932546 24920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:39.934468 24920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:39.934572 24920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:39.938354 24920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35605
I20260928 17:50:39.938391 24987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35605 every 8 connection(s)
I20260928 17:50:39.938692 24920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:39.939710 24988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:39.941880 24988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401: Bootstrap starting.
I20260928 17:50:39.942385 24988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:39.942602 24988 log.cc:824] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:39.942741 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 24920
I20260928 17:50:39.942847 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:39.943015 3960 external_mini_cluster.cc:1428] Setting key 5bb2345bdba85e6eed9ea0744393a964
I20260928 17:50:39.943267 24988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401: No bootstrap required, opened a new log
I20260928 17:50:39.945581 24988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ed743946a74e6c87a607d5343e1401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35605 } }
I20260928 17:50:39.945710 24988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:39.945732 24988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ed743946a74e6c87a607d5343e1401, State: Initialized, Role: FOLLOWER
I20260928 17:50:39.945884 24988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ed743946a74e6c87a607d5343e1401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35605 } }
I20260928 17:50:39.945963 24988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:39.946002 24988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:39.946043 24988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:39.946668 24988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ed743946a74e6c87a607d5343e1401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35605 } }
I20260928 17:50:39.946772 24988 leader_election.cc:302] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49ed743946a74e6c87a607d5343e1401; no voters:
I20260928 17:50:39.946918 24988 leader_election.cc:288] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:39.946941 24993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:39.947093 24993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [term 1 LEADER]: Becoming Leader. State: Replica: 49ed743946a74e6c87a607d5343e1401, State: Running, Role: LEADER
I20260928 17:50:39.947165 24988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:39.947172 24993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ed743946a74e6c87a607d5343e1401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35605 } }
I20260928 17:50:39.947579 24994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49ed743946a74e6c87a607d5343e1401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ed743946a74e6c87a607d5343e1401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35605 } } }
I20260928 17:50:39.947664 24994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:39.948158 24995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ed743946a74e6c87a607d5343e1401. Latest consensus state: current_term: 1 leader_uuid: "49ed743946a74e6c87a607d5343e1401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ed743946a74e6c87a607d5343e1401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35605 } } }
I20260928 17:50:39.948194 25002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:39.948302 24995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:39.948856 25002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:39.949239 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:39.943472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:54759 (local address 127.0.0.1:35605)
0928 17:50:39.943637 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:39.943645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:39.943669 (+ 24us) server_negotiation.cc:407] Connection header received
0928 17:50:39.944418 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:39.944425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:39.944472 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:39.944578 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:39.944802 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:39.945686 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:39.946712 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:39.947326 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:39.947465 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:39.948163 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:39.948167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:39.948168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:39.948367 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:39.948406 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:39.948421 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:39.948496 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:39.948817 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 17:50:39.948844 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260928 17:50:39.950196 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:39.950505 25002 catalog_manager.cc:1347] Generated new cluster ID: 3ac650a9ab4d4cb38df54a36f74f59af
I20260928 17:50:39.950553 25002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:39.955255 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:39.951794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:37075 (local address 127.0.0.1:35605)
0928 17:50:39.951847 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:39.951849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:39.951902 (+ 53us) server_negotiation.cc:407] Connection header received
0928 17:50:39.951977 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:39.951978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:39.951988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:39.952040 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:39.952183 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:39.952917 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:39.953869 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:39.954204 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:39.954274 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:39.954985 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:39.954987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:39.954987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:39.954992 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:39.955009 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:39.955014 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:39.955075 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:39.955139 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 17:50:39.955150 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:50:39.967557 25002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:39.968075 25002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:39.973004 25002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49ed743946a74e6c87a607d5343e1401: Generated new TSK 0
I20260928 17:50:39.973173 25002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:39.975392 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 24920
W20260928 17:50:39.979329 24905 connection.cc:535] client connection to 127.0.0.1:35605 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35605): Connection reset by peer (error 104)
2026-09-28T17:50:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:40 dist-test-slave-w3bm krb5kdc[25019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:40 dist-test-slave-w3bm krb5kdc[25019](info): set up 2 sockets
Sep 28 17:50:40 dist-test-slave-w3bm krb5kdc[25019](info): commencing operation
krb5kdc: starting...
W20260928 17:50:40.930902 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:41 dist-test-slave-w3bm krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:41.042790 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:41.119657 25032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:41.119848 25032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:41.119869 25032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:41.120920 25032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:41.120951 25032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:41.120965 25032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:41.120975 25032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:41.120985 25032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:41.122351 25032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:41.122520 25032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:41.122681 25032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:50:41.127249 25032 server_base.cc:1034] running on GCE node
W20260928 17:50:41.127197 25038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:41.127310 25040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:41.127204 25037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:41.127709 25032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:41.127985 25032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:41.129136 25032 hybrid_clock.cc:648] HybridClock initialized: now 1790617841129124 us; error 29 us; skew 500 ppm
Sep 28 17:50:41 dist-test-slave-w3bm krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:41.131494 25032 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:41.131906 25032 webserver.cc:460] Webserver started at http://127.0.0.1:39989/ using document root <none> and password file <none>
I20260928 17:50:41.132103 25032 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:41.132148 25032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:41.132265 25032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:41.133253 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "d47b2fd5690446b78826a12add4ce5c9"
format_stamp: "Formatted at 2026-09-28 17:50:41 on dist-test-slave-w3bm"
server_key: "1ab3bb0b5f7f695712634cb44de3414c"
server_key_iv: "51af344041c131d171825eaf116355d5"
server_key_version: "encryptionkey@0"
I20260928 17:50:41.133591 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "d47b2fd5690446b78826a12add4ce5c9"
format_stamp: "Formatted at 2026-09-28 17:50:41 on dist-test-slave-w3bm"
server_key: "1ab3bb0b5f7f695712634cb44de3414c"
server_key_iv: "51af344041c131d171825eaf116355d5"
server_key_version: "encryptionkey@0"
I20260928 17:50:41.134755 25032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 17:50:41.135402 25047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:41.135528 25032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:41.135599 25032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "d47b2fd5690446b78826a12add4ce5c9"
format_stamp: "Formatted at 2026-09-28 17:50:41 on dist-test-slave-w3bm"
server_key: "1ab3bb0b5f7f695712634cb44de3414c"
server_key_iv: "51af344041c131d171825eaf116355d5"
server_key_version: "encryptionkey@0"
I20260928 17:50:41.135661 25032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:41.138747 25032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:41.140722 25032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:41.140843 25032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:41.144449 25032 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39313
I20260928 17:50:41.144480 25099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39313 every 8 connection(s)
I20260928 17:50:41.144747 25032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:41.145610 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:41.146682 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 25032
I20260928 17:50:41.146780 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:41.146924 3960 external_mini_cluster.cc:1428] Setting key 309991217555437d3849669e67c96b66
I20260928 17:50:41.148236 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9: Bootstrap starting.
I20260928 17:50:41.148854 25100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:41.149153 25100 log.cc:824] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:41.149204 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:41.147405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:41409 (local address 127.0.0.1:39313)
0928 17:50:41.147583 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:41.147600 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:41.147633 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:50:41.148117 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:41.148125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:41.148158 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:41.148306 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:41.148791 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:41.148795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:41.148795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:41.148799 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:41.148840 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:41.148854 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:41.148942 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:41.148984 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 17:50:41.149029 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260928 17:50:41.149809 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9: No bootstrap required, opened a new log
I20260928 17:50:41.150048 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:41.152093 25100 raft_consensus.cc:348] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47b2fd5690446b78826a12add4ce5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39313 } }
I20260928 17:50:41.152238 25100 raft_consensus.cc:374] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:41.152315 25100 raft_consensus.cc:729] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47b2fd5690446b78826a12add4ce5c9, State: Initialized, Role: FOLLOWER
I20260928 17:50:41.152395 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:41.151502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:49423 (local address 127.0.0.1:39313)
0928 17:50:41.151542 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:41.151543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:41.151650 (+ 107us) server_negotiation.cc:407] Connection header received
0928 17:50:41.151677 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:41.151678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:41.151682 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:41.151751 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:41.152189 (+ 438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:41.152191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:41.152191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:41.152194 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:41.152207 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:41.152211 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:41.152269 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:41.152332 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 17:50:41.152344 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:50:41.152459 25100 consensus_queue.cc:260] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47b2fd5690446b78826a12add4ce5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39313 } }
I20260928 17:50:41.152552 25100 raft_consensus.cc:388] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:41.152599 25100 raft_consensus.cc:482] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:41.152640 25100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:41.153162 25100 raft_consensus.cc:504] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47b2fd5690446b78826a12add4ce5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39313 } }
I20260928 17:50:41.153244 25100 leader_election.cc:302] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d47b2fd5690446b78826a12add4ce5c9; no voters:
I20260928 17:50:41.153352 25100 leader_election.cc:288] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:41.153406 25110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:41.153548 25100 sys_catalog.cc:564] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:41.153565 25110 raft_consensus.cc:686] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [term 1 LEADER]: Becoming Leader. State: Replica: d47b2fd5690446b78826a12add4ce5c9, State: Running, Role: LEADER
I20260928 17:50:41.153700 25110 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47b2fd5690446b78826a12add4ce5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39313 } }
I20260928 17:50:41.154011 25112 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d47b2fd5690446b78826a12add4ce5c9. Latest consensus state: current_term: 1 leader_uuid: "d47b2fd5690446b78826a12add4ce5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47b2fd5690446b78826a12add4ce5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39313 } } }
I20260928 17:50:41.154234 25112 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:41.154168 25111 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d47b2fd5690446b78826a12add4ce5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47b2fd5690446b78826a12add4ce5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39313 } } }
I20260928 17:50:41.154439 25111 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:41.154577 25119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:41.155007 25119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:41.156488 25119 catalog_manager.cc:1347] Generated new cluster ID: 6970084aca3e459597dd19eb2f167f2e
I20260928 17:50:41.156536 25119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:41.164218 25119 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:41.164367 25119 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:41.167668 25119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d47b2fd5690446b78826a12add4ce5c9: Generated new TSK 0
I20260928 17:50:41.167835 25119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:41.170542 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 25032
2026-09-28T17:50:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:41 dist-test-slave-w3bm krb5kdc[25131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:41 dist-test-slave-w3bm krb5kdc[25131](info): set up 2 sockets
Sep 28 17:50:41 dist-test-slave-w3bm krb5kdc[25131](info): commencing operation
krb5kdc: starting...
W20260928 17:50:42.118965 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:42 dist-test-slave-w3bm krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:42.228430 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:42.307813 25144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:42.308049 25144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:42.308075 25144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:42.309182 25144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:42.309214 25144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:42.309227 25144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:42.309238 25144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:42.309249 25144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:42.310637 25144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:42.310818 25144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:42.310982 25144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:42.315544 25152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:42.315563 25149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:42.315626 25150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:42.315721 25144 server_base.cc:1034] running on GCE node
I20260928 17:50:42.316042 25144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:42.316323 25144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:42.317477 25144 hybrid_clock.cc:648] HybridClock initialized: now 1790617842317464 us; error 29 us; skew 500 ppm
Sep 28 17:50:42 dist-test-slave-w3bm krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:42.319792 25144 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:42.320214 25144 webserver.cc:460] Webserver started at http://127.0.0.1:34593/ using document root <none> and password file <none>
I20260928 17:50:42.320417 25144 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:42.320464 25144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:42.320583 25144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:42.321601 25144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "f150c52e6e9f4ab6ab651305e7152544"
format_stamp: "Formatted at 2026-09-28 17:50:42 on dist-test-slave-w3bm"
server_key: "c2b8aec734f324c196bc52feecd975bf"
server_key_iv: "3c72451b6bcc08c4127c9069c0949358"
server_key_version: "encryptionkey@0"
I20260928 17:50:42.321926 25144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "f150c52e6e9f4ab6ab651305e7152544"
format_stamp: "Formatted at 2026-09-28 17:50:42 on dist-test-slave-w3bm"
server_key: "c2b8aec734f324c196bc52feecd975bf"
server_key_iv: "3c72451b6bcc08c4127c9069c0949358"
server_key_version: "encryptionkey@0"
I20260928 17:50:42.323134 25144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:50:42.323844 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:42.323990 25144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:42.324070 25144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "f150c52e6e9f4ab6ab651305e7152544"
format_stamp: "Formatted at 2026-09-28 17:50:42 on dist-test-slave-w3bm"
server_key: "c2b8aec734f324c196bc52feecd975bf"
server_key_iv: "3c72451b6bcc08c4127c9069c0949358"
server_key_version: "encryptionkey@0"
I20260928 17:50:42.324131 25144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:42.342185 25144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:42.344074 25144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:42.344208 25144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:42.347888 25144 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42869
I20260928 17:50:42.347929 25211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42869 every 8 connection(s)
I20260928 17:50:42.348253 25144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:42.349129 25212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:42.351398 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544: Bootstrap starting.
I20260928 17:50:42.351940 25212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:42.352185 25212 log.cc:824] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:42.352550 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 25144
I20260928 17:50:42.352655 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:42.352821 3960 external_mini_cluster.cc:1428] Setting key e89284ed1ed90eebbc9678d4c6f35f95
I20260928 17:50:42.352890 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544: No bootstrap required, opened a new log
I20260928 17:50:42.354877 25212 raft_consensus.cc:348] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f150c52e6e9f4ab6ab651305e7152544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } }
I20260928 17:50:42.355046 25212 raft_consensus.cc:374] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:42.355078 25212 raft_consensus.cc:729] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f150c52e6e9f4ab6ab651305e7152544, State: Initialized, Role: FOLLOWER
I20260928 17:50:42.355204 25212 consensus_queue.cc:260] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f150c52e6e9f4ab6ab651305e7152544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } }
I20260928 17:50:42.355322 25212 raft_consensus.cc:388] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:42.355355 25212 raft_consensus.cc:482] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:42.355397 25212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:42.356035 25212 raft_consensus.cc:504] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f150c52e6e9f4ab6ab651305e7152544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } }
I20260928 17:50:42.356134 25212 leader_election.cc:302] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f150c52e6e9f4ab6ab651305e7152544; no voters:
I20260928 17:50:42.356269 25212 leader_election.cc:288] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:42.356305 25217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:42.356442 25217 raft_consensus.cc:686] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [term 1 LEADER]: Becoming Leader. State: Replica: f150c52e6e9f4ab6ab651305e7152544, State: Running, Role: LEADER
I20260928 17:50:42.356488 25212 sys_catalog.cc:564] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:42.356585 25217 consensus_queue.cc:237] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f150c52e6e9f4ab6ab651305e7152544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } }
W20260928 17:50:42.358134 25232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:42.358227 25232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:42.358563 25219 sys_catalog.cc:455] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f150c52e6e9f4ab6ab651305e7152544. Latest consensus state: current_term: 1 leader_uuid: "f150c52e6e9f4ab6ab651305e7152544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f150c52e6e9f4ab6ab651305e7152544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } } }
I20260928 17:50:42.358564 25218 sys_catalog.cc:455] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f150c52e6e9f4ab6ab651305e7152544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f150c52e6e9f4ab6ab651305e7152544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } } }
I20260928 17:50:42.358664 25219 sys_catalog.cc:458] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:42.358688 25218 sys_catalog.cc:458] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:42.358862 25234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:42.359035 25234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 17:50:42 dist-test-slave-w3bm krb5kdc[25131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 17:50:42.360817 25234 catalog_manager.cc:1347] Generated new cluster ID: 203aad6d3fa84a97a9579669ed2fde87
I20260928 17:50:42.360869 25234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:42.363844 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:42.353290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:38899 (local address 127.0.0.1:42869)
0928 17:50:42.353463 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:42.353473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:42.353510 (+ 37us) server_negotiation.cc:407] Connection header received
0928 17:50:42.354148 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:42.354158 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:42.354223 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:42.354379 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:42.354928 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:42.356537 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:42.357521 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:42.358103 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:42.358208 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:42.360773 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:42.360776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:42.360776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:42.360785 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:42.362309 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:42.362657 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:42.362660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:42.362660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:42.362752 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:42.362949 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:42.362949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:42.362950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:42.363166 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:42.363340 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:42.363466 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 17:50:42.363559 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260928 17:50:42.364689 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:42.372880 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:42.367495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:53057 (local address 127.0.0.1:42869)
0928 17:50:42.367537 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:42.367540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:42.367589 (+ 49us) server_negotiation.cc:407] Connection header received
0928 17:50:42.367659 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:42.367660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:42.367669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:42.367728 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:42.368178 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:42.368646 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:42.369340 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:42.369562 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:42.369615 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:42.370657 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:42.370659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:42.370659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:42.370666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:42.371860 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:42.372176 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:42.372177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:42.372177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:42.372229 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:42.372408 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:42.372409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:42.372409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:42.372509 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:42.372595 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:42.372736 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 17:50:42.372793 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:50:42.383877 25234 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:42.384385 25234 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:42.388712 25234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f150c52e6e9f4ab6ab651305e7152544: Generated new TSK 0
I20260928 17:50:42.388870 25234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:42.391825 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 25144
W20260928 17:50:42.395654 25129 connection.cc:535] client connection to 127.0.0.1:42869 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42869): Connection reset by peer (error 104)
2026-09-28T17:50:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:42 dist-test-slave-w3bm krb5kdc[25243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:42 dist-test-slave-w3bm krb5kdc[25243](info): set up 2 sockets
Sep 28 17:50:42 dist-test-slave-w3bm krb5kdc[25243](info): commencing operation
krb5kdc: starting...
W20260928 17:50:43.334962 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:43 dist-test-slave-w3bm krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:43.450019 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:43.527535 25256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:43.527736 25256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:43.527758 25256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:43.528882 25256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:43.528919 25256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:43.528932 25256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:43.528945 25256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:43.528968 25256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:43.530388 25256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:43.530596 25256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:43.530795 25256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:43.535375 25261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:43.535415 25262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:43.535382 25264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:43.535567 25256 server_base.cc:1034] running on GCE node
I20260928 17:50:43.535802 25256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:43.536074 25256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:43.537218 25256 hybrid_clock.cc:648] HybridClock initialized: now 1790617843537197 us; error 36 us; skew 500 ppm
Sep 28 17:50:43 dist-test-slave-w3bm krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:43.539510 25256 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:43.539913 25256 webserver.cc:460] Webserver started at http://127.0.0.1:35891/ using document root <none> and password file <none>
I20260928 17:50:43.540110 25256 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:43.540155 25256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:43.540272 25256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:43.541342 25256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "45e7d20644554ed1958b9306fbb83350"
format_stamp: "Formatted at 2026-09-28 17:50:43 on dist-test-slave-w3bm"
server_key: "f325e597e648a3bd1c810b12a44f1907"
server_key_iv: "5e5d297bb7bf9c4f5e7c05ef0377e359"
server_key_version: "encryptionkey@0"
I20260928 17:50:43.541663 25256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "45e7d20644554ed1958b9306fbb83350"
format_stamp: "Formatted at 2026-09-28 17:50:43 on dist-test-slave-w3bm"
server_key: "f325e597e648a3bd1c810b12a44f1907"
server_key_iv: "5e5d297bb7bf9c4f5e7c05ef0377e359"
server_key_version: "encryptionkey@0"
I20260928 17:50:43.542832 25256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:43.543506 25271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:43.543645 25256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:43.543725 25256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "45e7d20644554ed1958b9306fbb83350"
format_stamp: "Formatted at 2026-09-28 17:50:43 on dist-test-slave-w3bm"
server_key: "f325e597e648a3bd1c810b12a44f1907"
server_key_iv: "5e5d297bb7bf9c4f5e7c05ef0377e359"
server_key_version: "encryptionkey@0"
I20260928 17:50:43.543785 25256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:43.558847 25256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:43.560761 25256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:43.560899 25256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:43.564514 25256 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45051
I20260928 17:50:43.564563 25323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45051 every 8 connection(s)
I20260928 17:50:43.564878 25256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:43.565706 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:43.567909 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350: Bootstrap starting.
I20260928 17:50:43.568432 25324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:43.568674 25324 log.cc:824] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:43.569329 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350: No bootstrap required, opened a new log
I20260928 17:50:43.570844 25324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e7d20644554ed1958b9306fbb83350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45051 } }
I20260928 17:50:43.570976 25324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:43.570998 25324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e7d20644554ed1958b9306fbb83350, State: Initialized, Role: FOLLOWER
I20260928 17:50:43.571125 25324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e7d20644554ed1958b9306fbb83350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45051 } }
I20260928 17:50:43.571197 25324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:43.571255 25324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:43.571305 25324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:43.571779 25324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e7d20644554ed1958b9306fbb83350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45051 } }
I20260928 17:50:43.571859 25324 leader_election.cc:302] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45e7d20644554ed1958b9306fbb83350; no voters:
I20260928 17:50:43.571993 25324 leader_election.cc:288] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:43.572034 25327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:43.572189 25327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [term 1 LEADER]: Becoming Leader. State: Replica: 45e7d20644554ed1958b9306fbb83350, State: Running, Role: LEADER
I20260928 17:50:43.572216 25324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:43.572319 25327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e7d20644554ed1958b9306fbb83350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45051 } }
I20260928 17:50:43.572953 25328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45e7d20644554ed1958b9306fbb83350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e7d20644554ed1958b9306fbb83350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45051 } } }
I20260928 17:50:43.573045 25328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:43.573349 25329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e7d20644554ed1958b9306fbb83350. Latest consensus state: current_term: 1 leader_uuid: "45e7d20644554ed1958b9306fbb83350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e7d20644554ed1958b9306fbb83350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45051 } } }
I20260928 17:50:43.573413 25336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:43.573469 25329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:43.573879 25336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:43.574218 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 25256
I20260928 17:50:43.574327 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:43.574492 3960 external_mini_cluster.cc:1428] Setting key d90fcfbdcc62899736ab21388e65332d
I20260928 17:50:43.575412 25336 catalog_manager.cc:1347] Generated new cluster ID: 6f287bda25354e59bc4678e0a063eb0d
I20260928 17:50:43.575457 25336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:43.579377 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:43.574917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:37947 (local address 127.0.0.1:45051)
0928 17:50:43.575301 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:43.575311 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:43.575480 (+ 169us) server_negotiation.cc:407] Connection header received
0928 17:50:43.575609 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:43.575614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:43.575658 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:43.575766 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:43.575961 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:43.576875 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:43.577610 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:43.578028 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:43.578118 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:43.578747 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:43.578751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:43.578752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:43.578913 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:43.578942 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:43.578952 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:43.579018 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:43.579104 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 17:50:43.579130 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":63,"threads_started":1}
I20260928 17:50:43.580157 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:43.588281 25336 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:43.588945 25336 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:43.590785 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:43.588190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:54923 (local address 127.0.0.1:45051)
0928 17:50:43.588226 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:43.588228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:43.588377 (+ 149us) server_negotiation.cc:407] Connection header received
0928 17:50:43.588465 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:43.588466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:43.588476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:43.588518 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:43.588639 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:43.589111 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:43.589755 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:43.589961 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:43.590012 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:43.590571 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:43.590572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:43.590572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:43.590576 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:43.590590 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:43.590594 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:43.590639 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:43.590702 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 17:50:43.590714 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:50:43.609565 25336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45e7d20644554ed1958b9306fbb83350: Generated new TSK 0
I20260928 17:50:43.609782 25336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:43.614801 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 25256
W20260928 17:50:43.618455 25241 connection.cc:535] client connection to 127.0.0.1:45051 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45051): Connection reset by peer (error 104)
2026-09-28T17:50:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:43 dist-test-slave-w3bm krb5kdc[25355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:43 dist-test-slave-w3bm krb5kdc[25355](info): set up 2 sockets
Sep 28 17:50:43 dist-test-slave-w3bm krb5kdc[25355](info): commencing operation
krb5kdc: starting...
W20260928 17:50:44.582968 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:44 dist-test-slave-w3bm krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:50:44.694559 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:44.771458 25368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:44.771646 25368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:44.771667 25368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:44.772719 25368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:44.772751 25368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:44.772763 25368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:44.772774 25368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:44.772784 25368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:44.774117 25368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:44.774310 25368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:44.774466 25368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:44.778961 25376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:44.779085 25373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:44.778978 25368 server_base.cc:1034] running on GCE node
W20260928 17:50:44.779079 25374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:44.779497 25368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:44.779742 25368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:44.780892 25368 hybrid_clock.cc:648] HybridClock initialized: now 1790617844780876 us; error 28 us; skew 500 ppm
Sep 28 17:50:44 dist-test-slave-w3bm krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:44.804246 25368 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:50:44.804706 25368 webserver.cc:460] Webserver started at http://127.0.0.1:33591/ using document root <none> and password file <none>
I20260928 17:50:44.804896 25368 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:44.804935 25368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:44.805029 25368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:44.806041 25368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "77883f07b662442fb658aebae9f3c08e"
format_stamp: "Formatted at 2026-09-28 17:50:44 on dist-test-slave-w3bm"
server_key: "51a032f1e3e8f6a958caa837e483459f"
server_key_iv: "c3e8393589f683e0e94a6460cdca49b4"
server_key_version: "encryptionkey@0"
I20260928 17:50:44.806336 25368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "77883f07b662442fb658aebae9f3c08e"
format_stamp: "Formatted at 2026-09-28 17:50:44 on dist-test-slave-w3bm"
server_key: "51a032f1e3e8f6a958caa837e483459f"
server_key_iv: "c3e8393589f683e0e94a6460cdca49b4"
server_key_version: "encryptionkey@0"
I20260928 17:50:44.807561 25368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:50:44.808223 25383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:44.808388 25368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:50:44.808467 25368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "77883f07b662442fb658aebae9f3c08e"
format_stamp: "Formatted at 2026-09-28 17:50:44 on dist-test-slave-w3bm"
server_key: "51a032f1e3e8f6a958caa837e483459f"
server_key_iv: "c3e8393589f683e0e94a6460cdca49b4"
server_key_version: "encryptionkey@0"
I20260928 17:50:44.808532 25368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:44.811659 25368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:44.813364 25368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:44.813489 25368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:44.817152 25368 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37947
I20260928 17:50:44.817210 25435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37947 every 8 connection(s)
I20260928 17:50:44.817471 25368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:44.818338 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:44.818586 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 25368
I20260928 17:50:44.818686 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:44.818837 3960 external_mini_cluster.cc:1428] Setting key 7b8a18dbc9c2dc8372e0821dcea96fb5
I20260928 17:50:44.821027 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:44.819315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:50269 (local address 127.0.0.1:37947)
0928 17:50:44.819475 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:44.819483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:44.819506 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:50:44.819898 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:44.819904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:44.819929 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:44.820040 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:44.820609 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:44.820613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:44.820613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:44.820616 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:44.820650 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:44.820663 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:44.820734 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:44.820857 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 17:50:44.820900 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260928 17:50:44.821082 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e: Bootstrap starting.
I20260928 17:50:44.821772 25436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:44.821818 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:44.822036 25436 log.cc:824] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:44.822626 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e: No bootstrap required, opened a new log
I20260928 17:50:44.823997 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:44.823068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.23:40915 (local address 127.0.0.1:37947)
0928 17:50:44.823102 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:44.823103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:44.823178 (+ 75us) server_negotiation.cc:407] Connection header received
0928 17:50:44.823315 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:44.823316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:44.823320 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:44.823364 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:44.823813 (+ 449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:44.823814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:44.823815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:44.823819 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:44.823832 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:44.823836 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:44.823880 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:44.823928 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 17:50:44.823941 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:50:44.824641 25436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883f07b662442fb658aebae9f3c08e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260928 17:50:44.824767 25436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:44.824788 25436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77883f07b662442fb658aebae9f3c08e, State: Initialized, Role: FOLLOWER
I20260928 17:50:44.824900 25436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883f07b662442fb658aebae9f3c08e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260928 17:50:44.824975 25436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:44.825008 25436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:44.825055 25436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:44.825567 25436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883f07b662442fb658aebae9f3c08e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260928 17:50:44.825655 25436 leader_election.cc:302] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77883f07b662442fb658aebae9f3c08e; no voters:
I20260928 17:50:44.825793 25436 leader_election.cc:288] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:44.825841 25446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:44.826001 25436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:44.826009 25446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [term 1 LEADER]: Becoming Leader. State: Replica: 77883f07b662442fb658aebae9f3c08e, State: Running, Role: LEADER
I20260928 17:50:44.826179 25446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883f07b662442fb658aebae9f3c08e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260928 17:50:44.826517 25448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77883f07b662442fb658aebae9f3c08e. Latest consensus state: current_term: 1 leader_uuid: "77883f07b662442fb658aebae9f3c08e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883f07b662442fb658aebae9f3c08e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } } }
I20260928 17:50:44.826678 25447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77883f07b662442fb658aebae9f3c08e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883f07b662442fb658aebae9f3c08e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } } }
I20260928 17:50:44.826746 25448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:44.826828 25447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:44.827082 25457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:44.827524 25457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:44.829038 25457 catalog_manager.cc:1347] Generated new cluster ID: 640e88f9eaa340dc9488dd389b8588e7
I20260928 17:50:44.829077 25457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:44.837311 25457 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:44.837486 25457 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:44.843968 25457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77883f07b662442fb658aebae9f3c08e: Generated new TSK 0
I20260928 17:50:44.844138 25457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:44.848405 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 25368
2026-09-28T17:50:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:44 dist-test-slave-w3bm krb5kdc[25467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:44 dist-test-slave-w3bm krb5kdc[25467](info): set up 2 sockets
Sep 28 17:50:44 dist-test-slave-w3bm krb5kdc[25467](info): commencing operation
krb5kdc: starting...
W20260928 17:50:45.803030 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:45 dist-test-slave-w3bm krb5kdc[25467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:45.914139 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35757
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:35757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:45.990794 25480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:45.990988 25480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:45.991008 25480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:45.992144 25480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:45.992184 25480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:45.992197 25480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:45.992214 25480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:45.992237 25480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:45.993642 25480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:35757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:45.993857 25480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:45.994042 25480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:45.998514 25486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:45.998528 25485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:45.998668 25480 server_base.cc:1034] running on GCE node
W20260928 17:50:45.998620 25488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:45.998973 25480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:45.999274 25480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:46.000432 25480 hybrid_clock.cc:648] HybridClock initialized: now 1790617846000412 us; error 37 us; skew 500 ppm
Sep 28 17:50:45 dist-test-slave-w3bm krb5kdc[25467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617845, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:46.002601 25480 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:46.003018 25480 webserver.cc:460] Webserver started at http://127.3.222.62:45247/ using document root <none> and password file <none>
I20260928 17:50:46.003243 25480 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:46.003302 25480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:46.003432 25480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:46.004406 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "506da74323924631bf7c74b33eace066"
format_stamp: "Formatted at 2026-09-28 17:50:46 on dist-test-slave-w3bm"
server_key: "cd9c98d8ae7a4f02490feeaea2635f58"
server_key_iv: "77217ff6c3817e1df1ec51a70dd1896f"
server_key_version: "encryptionkey@0"
I20260928 17:50:46.004720 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "506da74323924631bf7c74b33eace066"
format_stamp: "Formatted at 2026-09-28 17:50:46 on dist-test-slave-w3bm"
server_key: "cd9c98d8ae7a4f02490feeaea2635f58"
server_key_iv: "77217ff6c3817e1df1ec51a70dd1896f"
server_key_version: "encryptionkey@0"
I20260928 17:50:46.005867 25480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:46.006490 25495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:46.006639 25480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:46.006707 25480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "506da74323924631bf7c74b33eace066"
format_stamp: "Formatted at 2026-09-28 17:50:46 on dist-test-slave-w3bm"
server_key: "cd9c98d8ae7a4f02490feeaea2635f58"
server_key_iv: "77217ff6c3817e1df1ec51a70dd1896f"
server_key_version: "encryptionkey@0"
I20260928 17:50:46.006770 25480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:46.013088 25480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:46.014635 25480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:46.014750 25480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:46.018347 25480 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:35757
I20260928 17:50:46.018370 25547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:35757 every 8 connection(s)
I20260928 17:50:46.018664 25480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:46.019477 25548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:46.021570 25548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066: Bootstrap starting.
I20260928 17:50:46.022091 25548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:46.022330 25548 log.cc:824] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:46.022939 25548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066: No bootstrap required, opened a new log
I20260928 17:50:46.024505 25548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506da74323924631bf7c74b33eace066" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35757 } }
I20260928 17:50:46.024637 25548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:46.024658 25548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506da74323924631bf7c74b33eace066, State: Initialized, Role: FOLLOWER
I20260928 17:50:46.024787 25548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506da74323924631bf7c74b33eace066" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35757 } }
I20260928 17:50:46.024844 25548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:46.024864 25548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:46.024909 25548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:46.025667 25548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506da74323924631bf7c74b33eace066" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35757 } }
I20260928 17:50:46.025753 25548 leader_election.cc:302] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 506da74323924631bf7c74b33eace066; no voters:
I20260928 17:50:46.025894 25548 leader_election.cc:288] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:46.025923 25551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:46.026051 25551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [term 1 LEADER]: Becoming Leader. State: Replica: 506da74323924631bf7c74b33eace066, State: Running, Role: LEADER
I20260928 17:50:46.026118 25548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:46.026156 25551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506da74323924631bf7c74b33eace066" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35757 } }
I20260928 17:50:46.026503 25552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "506da74323924631bf7c74b33eace066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506da74323924631bf7c74b33eace066" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35757 } } }
I20260928 17:50:46.026589 25552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:46.026703 25553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 506da74323924631bf7c74b33eace066. Latest consensus state: current_term: 1 leader_uuid: "506da74323924631bf7c74b33eace066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506da74323924631bf7c74b33eace066" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35757 } } }
I20260928 17:50:46.026850 25553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:46.027086 25560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:46.027520 25560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:46.027868 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 25480
I20260928 17:50:46.027983 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:46.028167 3960 external_mini_cluster.cc:1428] Setting key e7b6b2f2845065286325c48488497572
I20260928 17:50:46.031646 25560 catalog_manager.cc:1347] Generated new cluster ID: 0530290bd2364a0cae5c0402ff42ae54
I20260928 17:50:46.031699 25560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:50:46 dist-test-slave-w3bm krb5kdc[25467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:46.041060 25560 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:46.041646 25560 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:46.042495 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:46.028536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59687 (local address 127.3.222.62:35757)
0928 17:50:46.031685 (+ 3149us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:46.031694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:46.031726 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:50:46.031832 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:46.031839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:46.031894 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:46.032024 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:46.032769 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:46.034499 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:46.035554 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:46.036123 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:46.036216 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:46.038959 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:46.038965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:46.038965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:46.038977 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:46.040869 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:46.041309 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:46.041312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:46.041313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:46.041416 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:46.041636 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:46.041637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:46.041637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:46.041897 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:46.042038 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:46.042175 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 17:50:46.042270 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3113,"thread_start_us":60,"threads_started":1}
I20260928 17:50:46.043341 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:46.048794 25560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 506da74323924631bf7c74b33eace066: Generated new TSK 0
I20260928 17:50:46.048951 25560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:46.051195 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:46.044729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32847 (local address 127.3.222.62:35757)
0928 17:50:46.044771 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:46.044773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:46.044953 (+ 180us) server_negotiation.cc:407] Connection header received
0928 17:50:46.045002 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:46.045002 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:46.045011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:46.045058 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:46.045628 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:46.046365 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:46.047376 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:46.047695 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:46.047749 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:46.049088 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:46.049089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:46.049089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:46.049096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:46.050228 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:46.050603 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:46.050604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:46.050604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:46.050644 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:46.050805 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:46.050806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:46.050806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:46.050887 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:46.050966 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:46.051096 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 17:50:46.051139 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:50:46.052881 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 25480
W20260928 17:50:46.056806 25465 connection.cc:535] client connection to 127.3.222.62:35757 recv error: Network error: failed to read from TLS socket (remote: 127.3.222.62:35757): Connection reset by peer (error 104)
2026-09-28T17:50:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:46 dist-test-slave-w3bm krb5kdc[25579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:46 dist-test-slave-w3bm krb5kdc[25579](info): set up 2 sockets
Sep 28 17:50:46 dist-test-slave-w3bm krb5kdc[25579](info): commencing operation
krb5kdc: starting...
W20260928 17:50:46.998984 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:47 dist-test-slave-w3bm krb5kdc[25579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:47.110535 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:43791
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:47.188133 25592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:47.188338 25592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:47.188369 25592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:47.189488 25592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:47.189528 25592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:47.189545 25592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:47.189561 25592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:47.189575 25592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:47.190994 25592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:43791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:47.191242 25592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:47.191442 25592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:47.195951 25597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:47.195914 25598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:47.195901 25600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:47.196065 25592 server_base.cc:1034] running on GCE node
I20260928 17:50:47.196346 25592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:47.196583 25592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:47.197723 25592 hybrid_clock.cc:648] HybridClock initialized: now 1790617847197708 us; error 27 us; skew 500 ppm
Sep 28 17:50:47 dist-test-slave-w3bm krb5kdc[25579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617847, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:47.200058 25592 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:47.200531 25592 webserver.cc:460] Webserver started at http://127.3.222.62:43031/ using document root <none> and password file <none>
I20260928 17:50:47.200726 25592 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:47.200771 25592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:47.200893 25592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:47.201876 25592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "f794967daa37469eadbfe6940335176c"
format_stamp: "Formatted at 2026-09-28 17:50:47 on dist-test-slave-w3bm"
server_key: "894f00e7df6d6553957dd03a5ba805ae"
server_key_iv: "d70eb09e33fd23db3295eb39c02bf7c8"
server_key_version: "encryptionkey@0"
I20260928 17:50:47.202190 25592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "f794967daa37469eadbfe6940335176c"
format_stamp: "Formatted at 2026-09-28 17:50:47 on dist-test-slave-w3bm"
server_key: "894f00e7df6d6553957dd03a5ba805ae"
server_key_iv: "d70eb09e33fd23db3295eb39c02bf7c8"
server_key_version: "encryptionkey@0"
I20260928 17:50:47.203455 25592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:50:47.204154 25607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:47.204308 25592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:47.204416 25592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "f794967daa37469eadbfe6940335176c"
format_stamp: "Formatted at 2026-09-28 17:50:47 on dist-test-slave-w3bm"
server_key: "894f00e7df6d6553957dd03a5ba805ae"
server_key_iv: "d70eb09e33fd23db3295eb39c02bf7c8"
server_key_version: "encryptionkey@0"
I20260928 17:50:47.204476 25592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:47.213130 25592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:47.214856 25592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:47.214978 25592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:47.218640 25592 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:43791
I20260928 17:50:47.218673 25659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:43791 every 8 connection(s)
I20260928 17:50:47.218971 25592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:47.219782 25660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:47.221967 25660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c: Bootstrap starting.
I20260928 17:50:47.222501 25660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:47.222743 25660 log.cc:824] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:47.223392 25660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c: No bootstrap required, opened a new log
I20260928 17:50:47.224524 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 25592
I20260928 17:50:47.224612 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:47.224774 3960 external_mini_cluster.cc:1428] Setting key a3652acdf5474f79bf57fa1071822f84
I20260928 17:50:47.225046 25660 raft_consensus.cc:348] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f794967daa37469eadbfe6940335176c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43791 } }
I20260928 17:50:47.225283 25660 raft_consensus.cc:374] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:47.225376 25660 raft_consensus.cc:729] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f794967daa37469eadbfe6940335176c, State: Initialized, Role: FOLLOWER
I20260928 17:50:47.225589 25660 consensus_queue.cc:260] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f794967daa37469eadbfe6940335176c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43791 } }
I20260928 17:50:47.225667 25660 raft_consensus.cc:388] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:47.225698 25660 raft_consensus.cc:482] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:47.225744 25660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:47.226527 25660 raft_consensus.cc:504] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f794967daa37469eadbfe6940335176c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43791 } }
I20260928 17:50:47.226619 25660 leader_election.cc:302] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f794967daa37469eadbfe6940335176c; no voters:
I20260928 17:50:47.226768 25660 leader_election.cc:288] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:47.226797 25665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:47.226895 25665 raft_consensus.cc:686] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [term 1 LEADER]: Becoming Leader. State: Replica: f794967daa37469eadbfe6940335176c, State: Running, Role: LEADER
I20260928 17:50:47.227018 25660 sys_catalog.cc:564] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:47.227026 25665 consensus_queue.cc:237] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f794967daa37469eadbfe6940335176c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43791 } }
I20260928 17:50:47.227382 25667 sys_catalog.cc:455] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f794967daa37469eadbfe6940335176c. Latest consensus state: current_term: 1 leader_uuid: "f794967daa37469eadbfe6940335176c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f794967daa37469eadbfe6940335176c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43791 } } }
I20260928 17:50:47.227391 25666 sys_catalog.cc:455] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f794967daa37469eadbfe6940335176c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f794967daa37469eadbfe6940335176c" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 43791 } } }
I20260928 17:50:47.227550 25666 sys_catalog.cc:458] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:47.227540 25667 sys_catalog.cc:458] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c [sys.catalog]: This master's current role is: LEADER
W20260928 17:50:47.228718 25681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:47.228813 25681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:47.228919 25674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:47.229061 25674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:47.230173 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:47.225233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37871 (local address 127.3.222.62:43791)
0928 17:50:47.225522 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:47.225532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:47.225561 (+ 29us) server_negotiation.cc:407] Connection header received
0928 17:50:47.225657 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:47.225662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:47.225706 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:47.225827 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:47.226028 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:47.227022 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:47.228073 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:47.228663 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:47.228766 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:47.229520 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:47.229524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:47.229524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:47.229722 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:47.229751 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:47.229762 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:47.229845 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:47.229926 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 17:50:47.229954 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
I20260928 17:50:47.230564 25674 catalog_manager.cc:1347] Generated new cluster ID: 14df36700aad4e7c98d7d235a986a0d6
I20260928 17:50:47.230611 25674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:47.230948 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:47.234681 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:47.232175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59559 (local address 127.3.222.62:43791)
0928 17:50:47.232206 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:47.232208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:47.232310 (+ 102us) server_negotiation.cc:407] Connection header received
0928 17:50:47.232369 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:47.232370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:47.232379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:47.232428 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:47.232537 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:47.232983 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:47.233625 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:47.233856 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:47.233911 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:47.234473 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:47.234474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:47.234475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:47.234479 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:47.234492 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:47.234496 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:47.234539 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:47.234604 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 17:50:47.234614 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:50:47.250236 25674 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:47.250789 25674 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:47.255015 25674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f794967daa37469eadbfe6940335176c: Generated new TSK 0
I20260928 17:50:47.255180 25674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:47.263070 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 25592
W20260928 17:50:47.266779 25577 connection.cc:535] client connection to 127.3.222.62:43791 recv error: Network error: failed to read from TLS socket (remote: 127.3.222.62:43791): Connection reset by peer (error 104)
2026-09-28T17:50:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:47 dist-test-slave-w3bm krb5kdc[25691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:47 dist-test-slave-w3bm krb5kdc[25691](info): set up 2 sockets
Sep 28 17:50:47 dist-test-slave-w3bm krb5kdc[25691](info): commencing operation
krb5kdc: starting...
W20260928 17:50:48.211040 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:48 dist-test-slave-w3bm krb5kdc[25691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:48.327473 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41459
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:48.405972 25704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:48.406157 25704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:48.406179 25704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:48.407317 25704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:48.407356 25704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:48.407369 25704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:48.407380 25704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:48.407390 25704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:48.408788 25704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:48.408996 25704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:48.409186 25704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:48.413769 25712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:48.413784 25710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:48.413780 25709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:48.413944 25704 server_base.cc:1034] running on GCE node
I20260928 17:50:48.414219 25704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:48.414515 25704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:48.415647 25704 hybrid_clock.cc:648] HybridClock initialized: now 1790617848415632 us; error 31 us; skew 500 ppm
Sep 28 17:50:48 dist-test-slave-w3bm krb5kdc[25691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617848, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:48.417816 25704 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:48.418216 25704 webserver.cc:460] Webserver started at http://127.3.222.62:32793/ using document root <none> and password file <none>
I20260928 17:50:48.418398 25704 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:48.418452 25704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:48.418571 25704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:48.419600 25704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "ff2a25ac0e75450aa6494c880bd6f361"
format_stamp: "Formatted at 2026-09-28 17:50:48 on dist-test-slave-w3bm"
server_key: "d6aaf3ddda276f7605e01ac26ed0ce23"
server_key_iv: "560eb9e6a5b68b70ecb6a6e16acc7787"
server_key_version: "encryptionkey@0"
I20260928 17:50:48.419914 25704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "ff2a25ac0e75450aa6494c880bd6f361"
format_stamp: "Formatted at 2026-09-28 17:50:48 on dist-test-slave-w3bm"
server_key: "d6aaf3ddda276f7605e01ac26ed0ce23"
server_key_iv: "560eb9e6a5b68b70ecb6a6e16acc7787"
server_key_version: "encryptionkey@0"
I20260928 17:50:48.421072 25704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:48.421793 25719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:48.422000 25704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:50:48.422092 25704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "ff2a25ac0e75450aa6494c880bd6f361"
format_stamp: "Formatted at 2026-09-28 17:50:48 on dist-test-slave-w3bm"
server_key: "d6aaf3ddda276f7605e01ac26ed0ce23"
server_key_iv: "560eb9e6a5b68b70ecb6a6e16acc7787"
server_key_version: "encryptionkey@0"
I20260928 17:50:48.422154 25704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:48.425257 25704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:48.426985 25704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:48.427102 25704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:48.430616 25704 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:41459
I20260928 17:50:48.430646 25771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:41459 every 8 connection(s)
I20260928 17:50:48.430908 25704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:48.431782 25772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:48.432227 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 25704
I20260928 17:50:48.432380 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:48.432554 3960 external_mini_cluster.cc:1428] Setting key fc80d9f7f00d455c2fca30e844fae409
I20260928 17:50:48.434433 25772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361: Bootstrap starting.
I20260928 17:50:48.434650 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:48.433019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54567 (local address 127.3.222.62:41459)
0928 17:50:48.433154 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:48.433161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:48.433184 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:50:48.433555 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:48.433560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:48.433583 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:48.433687 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:48.434258 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:48.434261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:48.434262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:48.434265 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:48.434298 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:48.434308 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:48.434357 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:48.434452 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 17:50:48.434490 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
I20260928 17:50:48.435075 25772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:48.435374 25772 log.cc:824] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:48.435518 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:48.435982 25772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361: No bootstrap required, opened a new log
I20260928 17:50:48.437625 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:48.436824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60765 (local address 127.3.222.62:41459)
0928 17:50:48.436857 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:48.436858 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:48.436916 (+ 58us) server_negotiation.cc:407] Connection header received
0928 17:50:48.436988 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:48.436989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:48.436992 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:48.437033 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:48.437460 (+ 427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:48.437461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:48.437462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:48.437465 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:48.437474 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:48.437477 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:48.437514 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:48.437566 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 17:50:48.437575 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:50:48.438067 25772 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2a25ac0e75450aa6494c880bd6f361" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41459 } }
I20260928 17:50:48.438189 25772 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:48.438220 25772 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2a25ac0e75450aa6494c880bd6f361, State: Initialized, Role: FOLLOWER
I20260928 17:50:48.438329 25772 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2a25ac0e75450aa6494c880bd6f361" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41459 } }
I20260928 17:50:48.438396 25772 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:48.438424 25772 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:48.438468 25772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:48.438959 25772 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2a25ac0e75450aa6494c880bd6f361" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41459 } }
I20260928 17:50:48.439049 25772 leader_election.cc:302] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff2a25ac0e75450aa6494c880bd6f361; no voters:
I20260928 17:50:48.439184 25772 leader_election.cc:288] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:48.439249 25782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:48.439399 25782 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [term 1 LEADER]: Becoming Leader. State: Replica: ff2a25ac0e75450aa6494c880bd6f361, State: Running, Role: LEADER
I20260928 17:50:48.439417 25772 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:48.439519 25782 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2a25ac0e75450aa6494c880bd6f361" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41459 } }
I20260928 17:50:48.439837 25784 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff2a25ac0e75450aa6494c880bd6f361. Latest consensus state: current_term: 1 leader_uuid: "ff2a25ac0e75450aa6494c880bd6f361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2a25ac0e75450aa6494c880bd6f361" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41459 } } }
I20260928 17:50:48.439831 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff2a25ac0e75450aa6494c880bd6f361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2a25ac0e75450aa6494c880bd6f361" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41459 } } }
I20260928 17:50:48.439936 25784 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:48.439949 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:48.440142 25789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:48.440606 25789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:48.441923 25789 catalog_manager.cc:1347] Generated new cluster ID: 18b92afffae84a57826d16e95ac8540a
I20260928 17:50:48.441972 25789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:48.445724 25789 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:48.445868 25789 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:48.455340 25789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff2a25ac0e75450aa6494c880bd6f361: Generated new TSK 0
I20260928 17:50:48.455638 25789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:48.457914 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 25704
2026-09-28T17:50:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:48 dist-test-slave-w3bm krb5kdc[25803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:48 dist-test-slave-w3bm krb5kdc[25803](info): set up 2 sockets
Sep 28 17:50:48 dist-test-slave-w3bm krb5kdc[25803](info): commencing operation
krb5kdc: starting...
W20260928 17:50:49.407019 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:49 dist-test-slave-w3bm krb5kdc[25803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:49.515929 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:34345
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:34345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:49.592423 25816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:49.592620 25816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:49.592650 25816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:49.593742 25816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:49.593781 25816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:49.593799 25816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:49.593815 25816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:49.593840 25816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:49.595309 25816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:34345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:34345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:49.595538 25816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:49.595727 25816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:49.600091 25824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:49.600093 25822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:49.600114 25821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:49.600348 25816 server_base.cc:1034] running on GCE node
I20260928 17:50:49.600557 25816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:49.600806 25816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:49.601938 25816 hybrid_clock.cc:648] HybridClock initialized: now 1790617849601911 us; error 43 us; skew 500 ppm
Sep 28 17:50:49 dist-test-slave-w3bm krb5kdc[25803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617849, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:49.604171 25816 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:49.604564 25816 webserver.cc:460] Webserver started at http://127.3.222.62:37303/ using document root <none> and password file <none>
I20260928 17:50:49.604743 25816 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:49.604781 25816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:49.604861 25816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:49.605793 25816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "a5a7323b22c14766a21477460ebc796b"
format_stamp: "Formatted at 2026-09-28 17:50:49 on dist-test-slave-w3bm"
server_key: "5d4a89a6648b7ef93e6ae8e08deae856"
server_key_iv: "f8a0712ce0795abcfe61298f085adcf2"
server_key_version: "encryptionkey@0"
I20260928 17:50:49.606076 25816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "a5a7323b22c14766a21477460ebc796b"
format_stamp: "Formatted at 2026-09-28 17:50:49 on dist-test-slave-w3bm"
server_key: "5d4a89a6648b7ef93e6ae8e08deae856"
server_key_iv: "f8a0712ce0795abcfe61298f085adcf2"
server_key_version: "encryptionkey@0"
I20260928 17:50:49.607172 25816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:50:49.607841 25831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:49.607992 25816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:49.608063 25816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "a5a7323b22c14766a21477460ebc796b"
format_stamp: "Formatted at 2026-09-28 17:50:49 on dist-test-slave-w3bm"
server_key: "5d4a89a6648b7ef93e6ae8e08deae856"
server_key_iv: "f8a0712ce0795abcfe61298f085adcf2"
server_key_version: "encryptionkey@0"
I20260928 17:50:49.608122 25816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:49.616302 25816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:49.617820 25816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:49.617992 25816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:49.621585 25816 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:34345
I20260928 17:50:49.621624 25883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:34345 every 8 connection(s)
I20260928 17:50:49.621915 25816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:49.622699 25884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:49.624851 25884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b: Bootstrap starting.
I20260928 17:50:49.625346 25884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:49.625564 25884 log.cc:824] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:49.626149 25884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b: No bootstrap required, opened a new log
I20260928 17:50:49.627687 25884 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a7323b22c14766a21477460ebc796b" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34345 } }
I20260928 17:50:49.627816 25884 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:49.627836 25884 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a7323b22c14766a21477460ebc796b, State: Initialized, Role: FOLLOWER
I20260928 17:50:49.627950 25884 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a7323b22c14766a21477460ebc796b" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34345 } }
I20260928 17:50:49.628000 25884 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:49.628019 25884 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:49.628046 25884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:49.628768 25884 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a7323b22c14766a21477460ebc796b" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34345 } }
I20260928 17:50:49.628850 25884 leader_election.cc:302] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5a7323b22c14766a21477460ebc796b; no voters:
I20260928 17:50:49.628968 25884 leader_election.cc:288] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:49.629011 25887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:49.629174 25884 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:49.629179 25887 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [term 1 LEADER]: Becoming Leader. State: Replica: a5a7323b22c14766a21477460ebc796b, State: Running, Role: LEADER
I20260928 17:50:49.629326 25887 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a7323b22c14766a21477460ebc796b" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34345 } }
I20260928 17:50:49.629653 25889 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5a7323b22c14766a21477460ebc796b. Latest consensus state: current_term: 1 leader_uuid: "a5a7323b22c14766a21477460ebc796b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a7323b22c14766a21477460ebc796b" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34345 } } }
I20260928 17:50:49.629691 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 25816
I20260928 17:50:49.629783 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:49.629851 25889 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:49.629799 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5a7323b22c14766a21477460ebc796b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a7323b22c14766a21477460ebc796b" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 34345 } } }
I20260928 17:50:49.629973 3960 external_mini_cluster.cc:1428] Setting key 7760a38c4ea154d31440c2caa7c0c27c
I20260928 17:50:49.629987 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:49.630424 25896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:49.631246 25896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:49.632699 25896 catalog_manager.cc:1347] Generated new cluster ID: d3875477b554425e940404510c1da9fb
I20260928 17:50:49.632745 25896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:50:49 dist-test-slave-w3bm krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:49.641047 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:49.630377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54997 (local address 127.3.222.62:34345)
0928 17:50:49.630975 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:49.630983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:49.631007 (+ 24us) server_negotiation.cc:407] Connection header received
0928 17:50:49.631092 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:49.631097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:49.631138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:49.631273 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:49.631857 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:49.633093 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:49.634168 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:49.634760 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:49.634846 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:49.637468 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:49.637473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:49.637473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:49.637486 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:49.639543 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:49.639891 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:49.639893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:49.639894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:49.640020 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:49.640191 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:49.640192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:49.640192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:49.640492 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:49.640638 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:49.640705 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 17:50:49.640845 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":59,"threads_started":1}
I20260928 17:50:49.642066 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:49.648839 25896 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:49.649552 25896 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:49.650067 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:49.643529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43801 (local address 127.3.222.62:34345)
0928 17:50:49.643571 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:49.643573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:49.643582 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:50:49.643601 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:49.643602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:49.643610 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:49.643660 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:49.644232 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:49.644990 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:49.645992 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:49.646287 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:49.646337 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:49.647592 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:49.647594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:49.647594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:49.647600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:49.649110 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:49.649397 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:49.649398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:49.649398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:49.649465 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:49.649618 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:49.649619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:49.649619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:49.649717 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:49.649809 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:49.649900 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 17:50:49.649988 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:50:49.655771 25896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5a7323b22c14766a21477460ebc796b: Generated new TSK 0
I20260928 17:50:49.655936 25896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:49.664146 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 25816
2026-09-28T17:50:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:49 dist-test-slave-w3bm krb5kdc[25915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:49 dist-test-slave-w3bm krb5kdc[25915](info): set up 2 sockets
Sep 28 17:50:49 dist-test-slave-w3bm krb5kdc[25915](info): commencing operation
krb5kdc: starting...
W20260928 17:50:50.615020 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:50 dist-test-slave-w3bm krb5kdc[25915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:50.726298 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:40063
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:40063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:50.802815 25928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:50.803001 25928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:50.803031 25928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:50.804165 25928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:50.804207 25928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:50.804225 25928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:50.804241 25928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:50.804256 25928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:50.805662 25928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:40063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:40063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:50.805882 25928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:50.806071 25928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:50.810539 25933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:50.810612 25928 server_base.cc:1034] running on GCE node
W20260928 17:50:50.810595 25936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:50.810540 25934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:50.811044 25928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:50.811340 25928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:50.812476 25928 hybrid_clock.cc:648] HybridClock initialized: now 1790617850812470 us; error 36 us; skew 500 ppm
Sep 28 17:50:50 dist-test-slave-w3bm krb5kdc[25915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617850, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:50.819620 25928 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:50.820060 25928 webserver.cc:460] Webserver started at http://127.3.222.62:35941/ using document root <none> and password file <none>
I20260928 17:50:50.820266 25928 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:50.820312 25928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:50.820433 25928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:50.821427 25928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "1930da73a8b24902bd7144fa220a64f8"
format_stamp: "Formatted at 2026-09-28 17:50:50 on dist-test-slave-w3bm"
server_key: "5cf57457ac7a03fb546b4abf61052757"
server_key_iv: "670fe02a959e62c75550a6e236c52bea"
server_key_version: "encryptionkey@0"
I20260928 17:50:50.821743 25928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "1930da73a8b24902bd7144fa220a64f8"
format_stamp: "Formatted at 2026-09-28 17:50:50 on dist-test-slave-w3bm"
server_key: "5cf57457ac7a03fb546b4abf61052757"
server_key_iv: "670fe02a959e62c75550a6e236c52bea"
server_key_version: "encryptionkey@0"
I20260928 17:50:50.822980 25928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:50.823760 25943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:50.823913 25928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:50.823984 25928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "1930da73a8b24902bd7144fa220a64f8"
format_stamp: "Formatted at 2026-09-28 17:50:50 on dist-test-slave-w3bm"
server_key: "5cf57457ac7a03fb546b4abf61052757"
server_key_iv: "670fe02a959e62c75550a6e236c52bea"
server_key_version: "encryptionkey@0"
I20260928 17:50:50.824049 25928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:50.835855 25928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:50.837630 25928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:50.837754 25928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:50.841414 25928 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:40063
I20260928 17:50:50.841456 25995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:40063 every 8 connection(s)
I20260928 17:50:50.841750 25928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:50.842531 25996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:50.844657 25996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8: Bootstrap starting.
I20260928 17:50:50.845151 25996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:50.845369 25996 log.cc:824] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:50.845955 25996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8: No bootstrap required, opened a new log
I20260928 17:50:50.847430 25996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930da73a8b24902bd7144fa220a64f8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40063 } }
I20260928 17:50:50.847558 25996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:50.847576 25996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1930da73a8b24902bd7144fa220a64f8, State: Initialized, Role: FOLLOWER
I20260928 17:50:50.847680 25996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930da73a8b24902bd7144fa220a64f8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40063 } }
I20260928 17:50:50.847728 25996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:50.847746 25996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:50.847770 25996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:50.848467 25996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930da73a8b24902bd7144fa220a64f8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40063 } }
I20260928 17:50:50.848546 25996 leader_election.cc:302] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1930da73a8b24902bd7144fa220a64f8; no voters:
I20260928 17:50:50.848654 25996 leader_election.cc:288] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:50.848695 25999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:50.848865 25996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:50.848881 25999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1930da73a8b24902bd7144fa220a64f8, State: Running, Role: LEADER
I20260928 17:50:50.848958 25999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930da73a8b24902bd7144fa220a64f8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40063 } }
I20260928 17:50:50.849289 26000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1930da73a8b24902bd7144fa220a64f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930da73a8b24902bd7144fa220a64f8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40063 } } }
I20260928 17:50:50.849362 26000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:50.849551 26008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:50.849577 26001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1930da73a8b24902bd7144fa220a64f8. Latest consensus state: current_term: 1 leader_uuid: "1930da73a8b24902bd7144fa220a64f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930da73a8b24902bd7144fa220a64f8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 40063 } } }
I20260928 17:50:50.849824 26001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:50.850039 26008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:50.850250 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 25928
I20260928 17:50:50.850365 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:50.850553 3960 external_mini_cluster.cc:1428] Setting key 76df5e7d865029d17e4160954b2f0d7d
I20260928 17:50:50.851624 26008 catalog_manager.cc:1347] Generated new cluster ID: 858a49196d4541e28c6c86c2230c8b68
I20260928 17:50:50.851671 26008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:50.856297 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:50.851227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49951 (local address 127.3.222.62:40063)
0928 17:50:50.851350 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:50.851359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:50.851390 (+ 31us) server_negotiation.cc:407] Connection header received
0928 17:50:50.851497 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:50.851503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:50.851556 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:50.851693 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:50.851898 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:50.853219 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:50.854316 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:50.854908 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:50.854993 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:50.855645 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:50.855649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:50.855649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:50.855837 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:50.855882 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:50.855895 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:50.855950 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:50.856024 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 17:50:50.856081 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":62,"threads_started":1}
I20260928 17:50:50.857153 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:50.861732 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:50.858518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44601 (local address 127.3.222.62:40063)
0928 17:50:50.858551 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:50.858552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:50.858644 (+ 92us) server_negotiation.cc:407] Connection header received
0928 17:50:50.858715 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:50.858715 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:50.858723 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:50.858746 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:50.858895 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:50.859613 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:50.860556 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:50.860851 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:50.860884 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:50.861517 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:50.861518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:50.861518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:50.861523 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:50.861536 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:50.861540 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:50.861563 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:50.861643 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 17:50:50.861663 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:50:50.868732 26008 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:50.869187 26008 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:50.872360 26008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1930da73a8b24902bd7144fa220a64f8: Generated new TSK 0
I20260928 17:50:50.872515 26008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:50.880772 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 25928
2026-09-28T17:50:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:50 dist-test-slave-w3bm krb5kdc[26027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:50 dist-test-slave-w3bm krb5kdc[26027](info): set up 2 sockets
Sep 28 17:50:50 dist-test-slave-w3bm krb5kdc[26027](info): commencing operation
krb5kdc: starting...
W20260928 17:50:51.823189 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:51 dist-test-slave-w3bm krb5kdc[26027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:51.938531 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35917
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:35917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:52.015875 26040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:52.016070 26040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:52.016100 26040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:52.017203 26040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:52.017243 26040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:52.017261 26040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:52.017277 26040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:52.017292 26040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:52.018781 26040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:35917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:35917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:52.019007 26040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:52.019197 26040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:52.023743 26046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:52.023813 26048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:52.023782 26045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:52.023993 26040 server_base.cc:1034] running on GCE node
I20260928 17:50:52.024217 26040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:52.024454 26040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:52.025597 26040 hybrid_clock.cc:648] HybridClock initialized: now 1790617852025575 us; error 44 us; skew 500 ppm
Sep 28 17:50:52 dist-test-slave-w3bm krb5kdc[26027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617852, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:52.027966 26040 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:52.028375 26040 webserver.cc:460] Webserver started at http://127.3.222.62:34317/ using document root <none> and password file <none>
I20260928 17:50:52.028563 26040 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:52.028602 26040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:52.028689 26040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:52.029633 26040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "88a358a0bc3740e5bb8b1ea53639e195"
format_stamp: "Formatted at 2026-09-28 17:50:52 on dist-test-slave-w3bm"
server_key: "65108952cb43743777c640a022112df4"
server_key_iv: "e8f561e869069955b830f89300e59a46"
server_key_version: "encryptionkey@0"
I20260928 17:50:52.029907 26040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "88a358a0bc3740e5bb8b1ea53639e195"
format_stamp: "Formatted at 2026-09-28 17:50:52 on dist-test-slave-w3bm"
server_key: "65108952cb43743777c640a022112df4"
server_key_iv: "e8f561e869069955b830f89300e59a46"
server_key_version: "encryptionkey@0"
I20260928 17:50:52.031032 26040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:52.031718 26055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:52.031873 26040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:52.031955 26040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "88a358a0bc3740e5bb8b1ea53639e195"
format_stamp: "Formatted at 2026-09-28 17:50:52 on dist-test-slave-w3bm"
server_key: "65108952cb43743777c640a022112df4"
server_key_iv: "e8f561e869069955b830f89300e59a46"
server_key_version: "encryptionkey@0"
I20260928 17:50:52.032006 26040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:52.035082 26040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:52.037012 26040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:52.037110 26040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:52.040580 26040 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:35917
I20260928 17:50:52.040643 26107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:35917 every 8 connection(s)
I20260928 17:50:52.040894 26040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:52.041838 26108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:52.042371 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 26040
I20260928 17:50:52.042487 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:52.042655 3960 external_mini_cluster.cc:1428] Setting key 4f3aa378e1695e1d5dec6a8a083b07de
I20260928 17:50:52.044653 26108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195: Bootstrap starting.
I20260928 17:50:52.044948 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:52.043104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56569 (local address 127.3.222.62:35917)
0928 17:50:52.043343 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:52.043352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:52.043381 (+ 29us) server_negotiation.cc:407] Connection header received
0928 17:50:52.043880 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:52.043889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:52.043934 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:52.044073 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:52.044570 (+ 497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:52.044573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:52.044573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:52.044577 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:52.044616 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:52.044630 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:52.044698 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:52.044732 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 17:50:52.044768 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":58,"threads_started":1}
I20260928 17:50:52.045298 26108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:52.045575 26108 log.cc:824] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:52.045818 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:52.046166 26108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195: No bootstrap required, opened a new log
I20260928 17:50:52.047935 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:52.047123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38417 (local address 127.3.222.62:35917)
0928 17:50:52.047169 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:52.047171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:52.047178 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:50:52.047236 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:52.047236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:52.047240 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:52.047285 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:52.047755 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:52.047756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:52.047756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:50:52.047760 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:52.047771 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:50:52.047775 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:52.047818 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:52.047867 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 17:50:52.047879 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:50:52.048240 26108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a358a0bc3740e5bb8b1ea53639e195" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35917 } }
I20260928 17:50:52.048413 26108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:50:52.048441 26108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a358a0bc3740e5bb8b1ea53639e195, State: Initialized, Role: FOLLOWER
I20260928 17:50:52.048566 26108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a358a0bc3740e5bb8b1ea53639e195" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35917 } }
I20260928 17:50:52.048663 26118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:50:52.048738 26118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a358a0bc3740e5bb8b1ea53639e195" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35917 } }
I20260928 17:50:52.048648 26108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:50:52.048857 26108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:50:52.048846 26118 leader_election.cc:302] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88a358a0bc3740e5bb8b1ea53639e195; no voters:
I20260928 17:50:52.048897 26108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:52.049005 26118 leader_election.cc:288] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260928 17:50:52.049410 26108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a358a0bc3740e5bb8b1ea53639e195" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35917 } }
I20260928 17:50:52.049471 26108 leader_election.cc:302] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88a358a0bc3740e5bb8b1ea53639e195; no voters:
I20260928 17:50:52.049512 26119 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 17:50:52.049512 26108 leader_election.cc:288] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:50:52.049629 26118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:52.049710 26118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [term 1 LEADER]: Becoming Leader. State: Replica: 88a358a0bc3740e5bb8b1ea53639e195, State: Running, Role: LEADER
I20260928 17:50:52.049767 26108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:52.049783 26118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a358a0bc3740e5bb8b1ea53639e195" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35917 } }
I20260928 17:50:52.050089 26119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88a358a0bc3740e5bb8b1ea53639e195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a358a0bc3740e5bb8b1ea53639e195" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35917 } } }
I20260928 17:50:52.050108 26120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88a358a0bc3740e5bb8b1ea53639e195. Latest consensus state: current_term: 1 leader_uuid: "88a358a0bc3740e5bb8b1ea53639e195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a358a0bc3740e5bb8b1ea53639e195" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 35917 } } }
I20260928 17:50:52.050282 26120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:52.050217 26119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:52.050508 26127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:52.050864 26127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:52.052302 26127 catalog_manager.cc:1347] Generated new cluster ID: fe543fb6ceb04a99b885e90faaf03ecf
I20260928 17:50:52.052340 26127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:52.069325 26127 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:52.069545 26127 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:52.073814 26127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88a358a0bc3740e5bb8b1ea53639e195: Generated new TSK 0
I20260928 17:50:52.074002 26127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:52.083822 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 26040
2026-09-28T17:50:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1203 ms)
[----------] 18 tests from AuthTokenIssuingTest (21818 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:52 dist-test-slave-w3bm krb5kdc[26139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:52 dist-test-slave-w3bm krb5kdc[26139](info): set up 2 sockets
Sep 28 17:50:52 dist-test-slave-w3bm krb5kdc[26139](info): commencing operation
krb5kdc: starting...
W20260928 17:50:53.031064 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:53.146495 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41555
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41555,127.3.222.61:45377,127.3.222.60:42415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:53.223697 26152 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:50:53.223905 26152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:53.223929 26152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:53.223954 26152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:53.225054 26152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:53.225092 26152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:53.225111 26152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:53.225126 26152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:53.225142 26152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:53.226603 26152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41555,127.3.222.61:45377,127.3.222.60:42415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:41555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:53.226832 26152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:53.227025 26152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:50:53.231688 26152 server_base.cc:1034] running on GCE node
W20260928 17:50:53.231706 26158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:53.231839 26160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:53.231729 26157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:53.232105 26152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:53.232434 26152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:53.233601 26152 hybrid_clock.cc:648] HybridClock initialized: now 1790617853233589 us; error 26 us; skew 500 ppm
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:53.235975 26152 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:53.236337 26152 webserver.cc:460] Webserver started at http://127.3.222.62:35225/ using document root <none> and password file <none>
I20260928 17:50:53.236529 26152 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:53.236567 26152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:53.236653 26152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:53.237617 26152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "ffc29c1032584ea68c54cad3ce8641b8"
format_stamp: "Formatted at 2026-09-28 17:50:53 on dist-test-slave-w3bm"
server_key: "cce7cbbb52de19d1e4c13a8465cab32f"
server_key_iv: "5047e755195ad7f5f6d884f4a32be884"
server_key_version: "encryptionkey@0"
I20260928 17:50:53.237933 26152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "ffc29c1032584ea68c54cad3ce8641b8"
format_stamp: "Formatted at 2026-09-28 17:50:53 on dist-test-slave-w3bm"
server_key: "cce7cbbb52de19d1e4c13a8465cab32f"
server_key_iv: "5047e755195ad7f5f6d884f4a32be884"
server_key_version: "encryptionkey@0"
I20260928 17:50:53.239080 26152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:50:53.239836 26167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:53.239972 26152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:53.240056 26152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "ffc29c1032584ea68c54cad3ce8641b8"
format_stamp: "Formatted at 2026-09-28 17:50:53 on dist-test-slave-w3bm"
server_key: "cce7cbbb52de19d1e4c13a8465cab32f"
server_key_iv: "5047e755195ad7f5f6d884f4a32be884"
server_key_version: "encryptionkey@0"
I20260928 17:50:53.240108 26152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:53.250896 26152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:53.252689 26152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:53.252815 26152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:53.256508 26152 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:41555
I20260928 17:50:53.256992 26152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:53.257406 26219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:41555 every 8 connection(s)
I20260928 17:50:53.258213 26220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:53.260280 26220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:53.260942 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 26152
I20260928 17:50:53.261060 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:53.261252 3960 external_mini_cluster.cc:1428] Setting key e6cde19178f433fbceeb10ae4fe09905
WARNING: no policy specified for kudu/127.3.222.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.61@KRBTEST.COM" created.
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:53.277045 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.261136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41555 (local address 127.0.0.1:34204)
0928 17:50:53.261354 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.261403 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:53.261990 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:53.263647 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:53.263649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:53.263928 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:53.264076 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.264080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.265276 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.265277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.266296 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.266299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.267280 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.267282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.267355 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.269659 (+ 2304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:53.269667 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:53.273517 (+ 3850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:53.275592 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.275598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.275600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.275840 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.276069 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.276070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.276070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.276152 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.276472 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:53.276477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:53.276561 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 17:50:53.276640 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 17:50:53.276847 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":74,"thread_start_us":70,"threads_started":1}
I20260928 17:50:53.277050 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.261372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.3.222.62:41555)
0928 17:50:53.263387 (+ 2015us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.263392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.263418 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:50:53.263499 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.263501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.263530 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.263623 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.264167 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.265188 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.266381 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.266815 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.266937 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.273651 (+ 6714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.273661 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.273661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.273669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.275506 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.275922 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.275925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.275926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.276003 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.276203 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.276203 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.276204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.276412 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.276483 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.276681 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.276791 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1978,"thread_start_us":61,"threads_started":1}
I20260928 17:50:53.277886 26220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:53.278287 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.278140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45377 (local address 127.0.0.1:37196)
0928 17:50:53.278193 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.278234 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 17:50:53.278553 26170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
I20260928 17:50:53.278959 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.278873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45377 (local address 127.0.0.1:37206)
0928 17:50:53.278911 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.278935 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 17:50:53.279052 26220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.61:45377: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab.
Entry for principal kudu/127.3.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab.
I20260928 17:50:53.286310 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.61:45377
--webserver_interface=127.3.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab
--principal=kudu/127.3.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41555,127.3.222.61:45377,127.3.222.60:42415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:50:53.328295 26220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } attempt: 1
I20260928 17:50:53.328797 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.328670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45377 (local address 127.0.0.1:37222)
0928 17:50:53.328730 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.328764 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 17:50:53.329150 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.329069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45377 (local address 127.0.0.1:37224)
0928 17:50:53.329108 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.329131 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 17:50:53.329236 26220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.61:45377: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
W20260928 17:50:53.366560 26227 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:50:53.366755 26227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:53.366770 26227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:53.366788 26227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:53.367887 26227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:53.367924 26227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:53.367937 26227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:53.367949 26227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:53.367964 26227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:53.369352 26227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41555,127.3.222.61:45377,127.3.222.60:42415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab
--principal=kudu/127.3.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.61:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:53.369527 26227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:53.369686 26227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:53.374164 26232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:53.374258 26233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:53.374163 26235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:53.374315 26227 server_base.cc:1034] running on GCE node
I20260928 17:50:53.374604 26227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:53.374853 26227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:53.375986 26227 hybrid_clock.cc:648] HybridClock initialized: now 1790617853375969 us; error 27 us; skew 500 ppm
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:53.378249 26227 init.cc:377] Logged in from keytab as kudu/127.3.222.61@KRBTEST.COM (short username kudu)
I20260928 17:50:53.378650 26227 webserver.cc:460] Webserver started at http://127.3.222.61:32885/ using document root <none> and password file <none>
I20260928 17:50:53.378849 26227 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:53.378893 26227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:53.379010 26227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:53.380010 26227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/data/instance:
uuid: "4984db952199494e968ae63db5bafde7"
format_stamp: "Formatted at 2026-09-28 17:50:53 on dist-test-slave-w3bm"
server_key: "871243442ce43c4edba0d9642da09a86"
server_key_iv: "4edf01ea5dcb0b9034b614cc9ef0ad6e"
server_key_version: "encryptionkey@0"
I20260928 17:50:53.380335 26227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/wal/instance:
uuid: "4984db952199494e968ae63db5bafde7"
format_stamp: "Formatted at 2026-09-28 17:50:53 on dist-test-slave-w3bm"
server_key: "871243442ce43c4edba0d9642da09a86"
server_key_iv: "4edf01ea5dcb0b9034b614cc9ef0ad6e"
server_key_version: "encryptionkey@0"
I20260928 17:50:53.381479 26227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:53.382061 26242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:53.382205 26227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:53.382280 26227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/wal
uuid: "4984db952199494e968ae63db5bafde7"
format_stamp: "Formatted at 2026-09-28 17:50:53 on dist-test-slave-w3bm"
server_key: "871243442ce43c4edba0d9642da09a86"
server_key_iv: "4edf01ea5dcb0b9034b614cc9ef0ad6e"
server_key_version: "encryptionkey@0"
I20260928 17:50:53.382344 26227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:53.393576 26227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:53.395285 26227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:53.395406 26227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:53.397375 26220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } attempt: 2
I20260928 17:50:53.397784 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.397645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45377 (local address 127.0.0.1:37232)
0928 17:50:53.397709 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.397749 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 17:50:53.398078 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.397997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45377 (local address 127.0.0.1:37236)
0928 17:50:53.398028 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.398056 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 17:50:53.398198 26220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.61:45377: Network error: Client connection negotiation failed: client connection to 127.3.222.61:45377: connect: Connection refused (error 111)
I20260928 17:50:53.399201 26227 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.61:45377
I20260928 17:50:53.399259 26294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.61:45377 every 8 connection(s)
I20260928 17:50:53.399555 26227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
I20260928 17:50:53.400367 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 26227
I20260928 17:50:53.400449 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-1/wal/instance
I20260928 17:50:53.400463 26295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:53.400641 3960 external_mini_cluster.cc:1428] Setting key ad38696e06ce1664f18af34e078ab0ac
I20260928 17:50:53.402048 26295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.60@KRBTEST.COM" created.
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:53.412743 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.402670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.3.222.62:41555)
0928 17:50:53.402707 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.402709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.402926 (+ 217us) server_negotiation.cc:407] Connection header received
0928 17:50:53.403439 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.403440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.403454 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.403504 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.404076 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.404571 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.405524 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.405727 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.405781 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.410244 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.410245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.410246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.410253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.411517 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.411890 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.411892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.411892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.411954 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.412154 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.412155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.412155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.412282 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.412344 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.412559 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.412645 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:50:53.412866 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.402663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41555 (local address 127.0.0.1:34212)
0928 17:50:53.402818 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.402851 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:53.403348 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:53.403568 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:53.403572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:53.403807 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:53.403989 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.403993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.404653 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.404654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.405444 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.405446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.405803 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.405804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.405920 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.406345 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:53.406353 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:53.410156 (+ 3803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:53.411586 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.411592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.411593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.411813 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.412013 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.412014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.412014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.412104 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.412335 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:53.412338 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:53.412442 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 17:50:53.412516 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 17:50:53.412687 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260928 17:50:53.413364 26295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:50:53.421553 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.413600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45377 (local address 127.0.0.1:37252)
0928 17:50:53.413650 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.413662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:53.413706 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:53.413902 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:53.413903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:53.414037 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:53.414106 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.414108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.414908 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.414909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.415722 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.415725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.416172 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.416173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.416245 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.416548 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:53.416553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:53.419301 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:53.420804 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.420805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.420805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.420957 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.421121 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.421122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.421122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.421166 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.421356 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:53.421357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:53.421388 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 17:50:53.421431 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 17:50:53.421492 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 17:50:53.421646 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.413637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.3.222.61:45377)
0928 17:50:53.413768 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.413773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.413790 (+ 17us) server_negotiation.cc:407] Connection header received
0928 17:50:53.413811 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.413813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.413823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.413879 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.414167 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.414830 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.415793 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.416101 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.416152 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.419372 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.419379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.419379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.419385 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.420739 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.421017 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.421020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.421020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.421073 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.421198 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.421198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.421198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.421316 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.421387 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.421480 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.421544 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260928 17:50:53.422130 26295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:53.422428 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.422332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43408)
0928 17:50:53.422370 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.422413 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 17:50:53.422549 26246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
I20260928 17:50:53.422868 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.422797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43424)
0928 17:50:53.422832 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.422854 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 17:50:53.422930 26295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:42415: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab.
Entry for principal kudu/127.3.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab.
I20260928 17:50:53.425369 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.60:42415
--webserver_interface=127.3.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab
--principal=kudu/127.3.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:41555,127.3.222.61:45377,127.3.222.60:42415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:50:53.472154 26295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } attempt: 1
I20260928 17:50:53.472674 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.472523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43438)
0928 17:50:53.472594 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.472640 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 17:50:53.473006 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.472915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43452)
0928 17:50:53.472958 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.472985 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 17:50:53.473093 26295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:42415: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
I20260928 17:50:53.489336 26220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } attempt: 3
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:50:53.495944 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.489660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45377 (local address 127.0.0.1:37268)
0928 17:50:53.489718 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.489729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:53.489778 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:53.489913 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:53.489913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:53.490053 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:53.490120 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.490122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.490806 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.490808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.491368 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.491371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.491739 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.491740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.491811 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.492266 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:53.492271 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:53.493595 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:53.495018 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.495019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.495020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.495192 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.495378 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.495379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.495379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.495434 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.495652 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:53.495653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:53.495699 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 17:50:53.495757 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 17:50:53.495849 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 17:50:53.495944 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.489721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.3.222.61:45377)
0928 17:50:53.489779 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.489781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.489790 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:50:53.489836 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.489837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.489847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.489891 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.490192 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.490725 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.491447 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.491678 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.491735 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.493674 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.493675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.493675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.493682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.494932 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.495270 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.495271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.495271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.495324 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.495483 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.495483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.495484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.495596 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.495651 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.495784 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.495854 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:50:53.496476 26220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:53.496791 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.496700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43462)
0928 17:50:53.496738 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.496764 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 17:50:53.497090 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.497019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43466)
0928 17:50:53.497057 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.497077 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 17:50:53.497164 26220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:42415: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
W20260928 17:50:53.503531 26302 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:50:53.503731 26302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:53.503746 26302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:53.503764 26302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:53.504853 26302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:53.504884 26302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:53.504896 26302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:53.504906 26302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:53.504917 26302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:53.506296 26302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:41555,127.3.222.61:45377,127.3.222.60:42415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab
--principal=kudu/127.3.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.60:42415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:53.506481 26302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:53.506651 26302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:53.511166 26310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:53.511133 26308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:53.511144 26307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:53.511353 26302 server_base.cc:1034] running on GCE node
I20260928 17:50:53.511685 26302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:53.511955 26302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:53.513109 26302 hybrid_clock.cc:648] HybridClock initialized: now 1790617853513092 us; error 30 us; skew 500 ppm
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:53.515463 26302 init.cc:377] Logged in from keytab as kudu/127.3.222.60@KRBTEST.COM (short username kudu)
I20260928 17:50:53.515868 26302 webserver.cc:460] Webserver started at http://127.3.222.60:34457/ using document root <none> and password file <none>
I20260928 17:50:53.516038 26302 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:53.516077 26302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:53.516165 26302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:53.517140 26302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/data/instance:
uuid: "58db5aea4e434c49b0419fe79b7311c8"
format_stamp: "Formatted at 2026-09-28 17:50:53 on dist-test-slave-w3bm"
server_key: "86e783a3efcae360f598d7750018bbb2"
server_key_iv: "59d826bb424af272bca555f8955f3e40"
server_key_version: "encryptionkey@0"
I20260928 17:50:53.517417 26302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/wal/instance:
uuid: "58db5aea4e434c49b0419fe79b7311c8"
format_stamp: "Formatted at 2026-09-28 17:50:53 on dist-test-slave-w3bm"
server_key: "86e783a3efcae360f598d7750018bbb2"
server_key_iv: "59d826bb424af272bca555f8955f3e40"
server_key_version: "encryptionkey@0"
I20260928 17:50:53.518543 26302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:53.519179 26317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:53.519356 26302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:53.519425 26302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/wal
uuid: "58db5aea4e434c49b0419fe79b7311c8"
format_stamp: "Formatted at 2026-09-28 17:50:53 on dist-test-slave-w3bm"
server_key: "86e783a3efcae360f598d7750018bbb2"
server_key_iv: "59d826bb424af272bca555f8955f3e40"
server_key_version: "encryptionkey@0"
I20260928 17:50:53.519495 26302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:53.527933 26302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:53.528307 26220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } attempt: 1
I20260928 17:50:53.528676 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.528547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43482)
0928 17:50:53.528604 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.528642 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:50:53.529011 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.528924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43492)
0928 17:50:53.528961 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.528987 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 17:50:53.529109 26220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:42415: Network error: Client connection negotiation failed: client connection to 127.3.222.60:42415: connect: Connection refused (error 111)
I20260928 17:50:53.529970 26302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:53.530093 26302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:53.533674 26302 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.60:42415
I20260928 17:50:53.533702 26369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.60:42415 every 8 connection(s)
I20260928 17:50:53.534013 26302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
I20260928 17:50:53.534937 26370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:53.536442 26370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:53.539513 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 26302
I20260928 17:50:53.539626 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617599653609-3960-0/minicluster-data/master-2/wal/instance
I20260928 17:50:53.539808 3960 external_mini_cluster.cc:1428] Setting key accda989c5e0c94adfb2fd5f2a329198
I20260928 17:50:53.541227 26295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } attempt: 2
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
I20260928 17:50:53.545538 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.537024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.3.222.62:41555)
0928 17:50:53.537093 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.537095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.537258 (+ 163us) server_negotiation.cc:407] Connection header received
0928 17:50:53.537707 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.537707 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.537716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.537751 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.538276 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.538763 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.539972 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.540246 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.540305 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.542707 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.542708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.542708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.542715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.544225 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.544640 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.544642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.544642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.544706 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.544957 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.544958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.544958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.545079 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.545134 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.545378 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.545454 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:50:53.545687 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.537001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:41555 (local address 127.0.0.1:34216)
0928 17:50:53.537151 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.537186 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:53.537625 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:53.537809 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:53.537812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:53.538026 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:53.538212 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.538216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.538842 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.538843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.539896 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.539899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.540344 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.540345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.540490 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.540990 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:53.540998 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:53.542634 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:53.544283 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.544290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.544291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.544552 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.544774 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.544775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.544775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.544901 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.545136 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:53.545141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:53.545265 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 17:50:53.545346 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 17:50:53.545475 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
I20260928 17:50:53.546149 26370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:53.548887 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.541450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43504)
0928 17:50:53.541497 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.541505 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:53.541539 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:53.541860 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:53.541862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:53.542024 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:53.542073 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.542075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.542942 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.542943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.543661 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.543665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.544077 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.544078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.544141 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.544444 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:53.544449 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:53.546168 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:53.547873 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.547875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.547875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.548071 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.548266 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.548267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.548267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.548339 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.548592 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:53.548593 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:53.548643 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 17:50:53.548705 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:50:53.548795 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":2}
I20260928 17:50:53.549180 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.541604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.3.222.60:42415)
0928 17:50:53.541734 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.541735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.541743 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:50:53.541760 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.541760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.541769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.541832 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.542183 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.542864 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.543727 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.544005 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.544073 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.546244 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.546250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.546250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.546256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.547784 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.548151 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.548153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.548154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.548213 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.548400 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.548400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.548401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.548523 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.548600 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.549009 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.549088 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:50:53.550575 26295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7: Bootstrap starting.
I20260928 17:50:53.551636 26295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:53.551931 26295 log.cc:824] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:53.552757 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.540212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.3.222.60:42415)
0928 17:50:53.540391 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.540395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.540411 (+ 16us) server_negotiation.cc:407] Connection header received
0928 17:50:53.540430 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.540432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.540451 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.540520 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.541175 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.542077 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.542971 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.543396 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.543467 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.546401 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.546402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.546403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.546409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.549005 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.549354 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.549355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.549355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.549400 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.552199 (+ 2799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.552201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.552201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.552323 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.552463 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.552547 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.552621 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260928 17:50:53.552776 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.546353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:45377 (local address 127.0.0.1:37276)
0928 17:50:53.546382 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.546391 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:53.546426 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:53.546600 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:53.546600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:53.546710 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:53.546757 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.546759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.547579 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.547580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.548284 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.548285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.548694 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.548695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.548762 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.549236 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:53.549240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:53.550294 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:53.551825 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.551826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.551826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.551966 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.552162 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.552163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.552163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.552225 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.552485 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:53.552486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:53.552524 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 17:50:53.552591 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:50:53.552666 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 17:50:53.552832 26295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7: No bootstrap required, opened a new log
I20260928 17:50:53.553006 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.546442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.3.222.61:45377)
0928 17:50:53.546483 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.546484 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.546491 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:50:53.546509 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.546510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.546518 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.546561 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.546834 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.547498 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.548336 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.548623 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.548694 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.550372 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.550373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.550374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.550380 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.551745 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.552036 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.552037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.552038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.552093 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.552277 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.552278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.552278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.552422 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.552483 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.552852 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.552927 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":22}
I20260928 17:50:53.553485 26370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:53.554507 26295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } }
I20260928 17:50:53.554613 26295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4984db952199494e968ae63db5bafde7, State: Initialized, Role: FOLLOWER
I20260928 17:50:53.554668 26295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } }
I20260928 17:50:53.554920 26377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.554982 26295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:53.555001 26377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:50:53.556638 26388 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:53.556690 26388 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
I20260928 17:50:53.556951 26227 master_runner.cc:186] Error getting master registration for 127.3.222.62:41555: OK, instance_id { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" instance_seqno: 1790617853255392 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:50:53.559695 26227 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:50:53.560360 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.553734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43518)
0928 17:50:53.553798 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.553807 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:53.553846 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:53.553947 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:53.553948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:53.554082 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:53.554114 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.554116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.554751 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.554752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.555466 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.555469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.555899 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.555900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.555957 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.556390 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:53.556394 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:53.557698 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:53.559354 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.559355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.559355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.559533 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.559773 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.559774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.559774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.559829 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.560068 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:53.560069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:53.560105 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 17:50:53.560165 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 17:50:53.560255 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 17:50:53.560401 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.553764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.3.222.60:42415)
0928 17:50:53.553792 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.553793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.553834 (+ 41us) server_negotiation.cc:407] Connection header received
0928 17:50:53.553895 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.553896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.553903 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.553943 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.554174 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.554617 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.555557 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.555834 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.555899 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.557778 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.557779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.557780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.557785 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.559276 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.559600 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.559600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.559601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.559644 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.559884 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.559886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.559886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.560003 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.560061 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.560192 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.560302 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":14}
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:53.561627 26370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8: Bootstrap starting.
I20260928 17:50:53.562157 26370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:53.562390 26370 log.cc:824] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:53.563076 26370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8: No bootstrap required, opened a new log
I20260928 17:50:53.563871 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.556758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.3.222.62:41555)
0928 17:50:53.556793 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.556795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.556801 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:50:53.556819 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.556819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.556827 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.556870 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.557464 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.558142 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.559074 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.559372 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.559431 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.561682 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.561683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.561684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.561690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.562892 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.563227 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.563228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.563228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.563282 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.563445 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.563446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.563446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.563548 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.563691 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.563725 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.563789 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:50:53.565032 26370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } }
I20260928 17:50:53.565191 26370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58db5aea4e434c49b0419fe79b7311c8, State: Initialized, Role: FOLLOWER
I20260928 17:50:53.565271 26370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } }
I20260928 17:50:53.565547 26370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 17:50:53.567327 26402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:53.567378 26402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:53.567613 26391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.567701 26391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:50:53.567427 26302 master_runner.cc:186] Error getting master registration for 127.3.222.62:41555: OK, instance_id { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" instance_seqno: 1790617853255392 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:50:53.569629 26302 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:50:53.572079 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.564523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.3.222.61:45377)
0928 17:50:53.564558 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.564560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.564596 (+ 36us) server_negotiation.cc:407] Connection header received
0928 17:50:53.564659 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.564660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.564668 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.564707 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.565108 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.565675 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.566405 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.566682 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.566740 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.569353 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.569355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.569355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.569363 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.570830 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.571316 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.571318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.571318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.571382 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.571581 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.571582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.571582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.571719 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.571869 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.571910 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.571990 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:50:53.573354 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:53.574604 26194 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ffc29c1032584ea68c54cad3ce8641b8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34224
I20260928 17:50:53.604197 26220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } attempt: 2
Sep 28 17:50:53 dist-test-slave-w3bm krb5kdc[26139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
I20260928 17:50:53.612365 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.604605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:42415 (local address 127.0.0.1:43534)
0928 17:50:53.604673 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:53.604684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:53.604726 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:53.604917 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:53.604919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:53.605121 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:53.605178 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.605180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.606110 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.606111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.606927 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:53.606931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.607425 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.607425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:53.607487 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.607953 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:53.607958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:53.609508 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:53.611301 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.611303 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.611303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.611521 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.611728 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:53.611729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:53.611729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:53.611788 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:53.612043 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:53.612044 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:53.612099 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 17:50:53.612164 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 17:50:53.612259 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 17:50:53.612368 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.604670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.3.222.60:42415)
0928 17:50:53.604714 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.604715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.604726 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:50:53.604806 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.604807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.604819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.604863 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.605265 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.606023 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.607018 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.607342 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.607425 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.609610 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.609612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.609612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.609619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.611187 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.611591 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.611592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.611593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.611655 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.611843 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.611844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.611845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.611972 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.612042 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.612193 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.612272 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:50:53.614331 26220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8: Bootstrap starting.
I20260928 17:50:53.615087 26220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:53.615434 26220 log.cc:824] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:53.616304 26220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8: No bootstrap required, opened a new log
I20260928 17:50:53.617632 26220 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } }
I20260928 17:50:53.617779 26220 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc29c1032584ea68c54cad3ce8641b8, State: Initialized, Role: FOLLOWER
I20260928 17:50:53.617866 26220 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } }
I20260928 17:50:53.618136 26405 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.618192 26220 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:53.618244 26194 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 17:50:53.618287 26405 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:53.618325 26194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:53.619158 26194 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } }
I20260928 17:50:53.619416 26194 leader_election.cc:288] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [CANDIDATE]: Term 1 election: Requested vote from peers 4984db952199494e968ae63db5bafde7 (127.3.222.61:45377), 58db5aea4e434c49b0419fe79b7311c8 (127.3.222.60:42415)
I20260928 17:50:53.619813 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffc29c1032584ea68c54cad3ce8641b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4984db952199494e968ae63db5bafde7"
I20260928 17:50:53.619956 26269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:53.620791 26269 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc29c1032584ea68c54cad3ce8641b8 in term 1.
I20260928 17:50:53.621008 26170 leader_election.cc:302] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4984db952199494e968ae63db5bafde7, ffc29c1032584ea68c54cad3ce8641b8; no voters:
I20260928 17:50:53.621302 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffc29c1032584ea68c54cad3ce8641b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "58db5aea4e434c49b0419fe79b7311c8"
I20260928 17:50:53.621413 26344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:53.622275 26344 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc29c1032584ea68c54cad3ce8641b8 in term 1.
I20260928 17:50:53.623373 26405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:53.623399 26152 master_runner.cc:263] No leader master found from master ffc29c1032584ea68c54cad3ce8641b8
I20260928 17:50:53.623452 26152 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:50:53.623515 26405 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [term 1 LEADER]: Becoming Leader. State: Replica: ffc29c1032584ea68c54cad3ce8641b8, State: Running, Role: LEADER
I20260928 17:50:53.623605 26405 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } }
W20260928 17:50:53.624585 26416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:53.624639 26416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:53.624689 26405 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc29c1032584ea68c54cad3ce8641b8. Latest consensus state: current_term: 1 leader_uuid: "ffc29c1032584ea68c54cad3ce8641b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.624748 26405 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:53.624912 26424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:53.625115 26424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:53.626493 26344 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [term 1 FOLLOWER]: Refusing update from remote peer ffc29c1032584ea68c54cad3ce8641b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:53.627085 26269 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [term 1 FOLLOWER]: Refusing update from remote peer ffc29c1032584ea68c54cad3ce8641b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:53.627405 26405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:53.628553 26391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc29c1032584ea68c54cad3ce8641b8. Latest consensus state: current_term: 1 leader_uuid: "ffc29c1032584ea68c54cad3ce8641b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.628693 26391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:53.628793 26405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:53.629841 26377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc29c1032584ea68c54cad3ce8641b8. Latest consensus state: current_term: 1 leader_uuid: "ffc29c1032584ea68c54cad3ce8641b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.629935 26377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:53.630798 26391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffc29c1032584ea68c54cad3ce8641b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.630921 26391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:53.631428 26405 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffc29c1032584ea68c54cad3ce8641b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.631553 26405 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:53.631664 26426 mvcc.cc:205] Tried to move back new op lower bound from 7334370728451956736 to 7334370728442990592. Current Snapshot: MvccSnapshot[applied={T|T < 7334370728451956736}]
I20260928 17:50:53.632129 26428 mvcc.cc:205] Tried to move back new op lower bound from 7334370728451956736 to 7334370728442990592. Current Snapshot: MvccSnapshot[applied={T|T < 7334370728451956736}]
I20260928 17:50:53.636581 26424 catalog_manager.cc:1347] Generated new cluster ID: 5657d66f8a1247f49bc08913b4ecd0ce
I20260928 17:50:53.636633 26424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:53.642839 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.628632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.3.222.61:45377)
0928 17:50:53.628671 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.628672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.635390 (+ 6718us) server_negotiation.cc:407] Connection header received
0928 17:50:53.635479 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.635480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.635490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.635517 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.636290 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.637004 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.638112 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.638397 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.638519 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.640059 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.640060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.640061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.640068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.641607 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.642022 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.642023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.642023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.642078 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.642282 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.642283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.642284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.642401 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.642480 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.642682 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.642754 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 17:50:53.645903 26248 tablet.cc:2369] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:50:53.647253 26377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffc29c1032584ea68c54cad3ce8641b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.647336 26377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:53.647820 26405 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffc29c1032584ea68c54cad3ce8641b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc29c1032584ea68c54cad3ce8641b8" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 41555 } } peers { permanent_uuid: "4984db952199494e968ae63db5bafde7" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 45377 } } peers { permanent_uuid: "58db5aea4e434c49b0419fe79b7311c8" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 42415 } } }
I20260928 17:50:53.647962 26405 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8 [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:53.648890 26424 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:53.649601 26424 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:53.653227 26432 mvcc.cc:205] Tried to move back new op lower bound from 7334370728451956736 to 7334370728442990592. Current Snapshot: MvccSnapshot[applied={T|T < 7334370728451956736}]
I20260928 17:50:53.655583 26424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc29c1032584ea68c54cad3ce8641b8: Generated new TSK 0
I20260928 17:50:53.656361 26424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:50:53.662566 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.628926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.3.222.62:41555)
0928 17:50:53.628958 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.628960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.643829 (+ 14869us) server_negotiation.cc:407] Connection header received
0928 17:50:53.643915 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.643916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.643927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.643955 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.644478 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.645149 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.654004 (+ 8855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.654290 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.654340 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.657774 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.657776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.657776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.657783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.659458 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.660705 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.660706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.660706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.660772 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.661937 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.661938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.661938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.662076 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.662160 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.662362 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.662478 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:50:53.667534 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:53.625027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.3.222.60:42415)
0928 17:50:53.625071 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.625073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.635365 (+ 10292us) server_negotiation.cc:407] Connection header received
0928 17:50:53.635429 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.635430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.635441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.635499 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.655979 (+ 20480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.656677 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.659855 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:53.660142 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:53.660187 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:53.663146 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:53.663147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:53.663148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:53.663155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:53.664801 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.665220 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.665221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.665221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.665296 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:53.666817 (+ 1521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:53.666818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:53.666819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:53.666955 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:53.667037 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:53.667351 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 17:50:53.667441 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 17:50:53.679637 26374 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.678110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.3.222.60:42415)
0928 17:50:53.678151 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.678153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.679021 (+ 868us) server_negotiation.cc:407] Connection header received
0928 17:50:53.679113 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.679114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.679123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.679150 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.679577 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43556: BlockingRecv error: recv got EOF from 127.0.0.1:43556 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:50:53.680450 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.678473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.3.222.61:45377)
0928 17:50:53.678512 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.678514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.679973 (+ 1459us) server_negotiation.cc:407] Connection header received
0928 17:50:53.680059 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.680061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.680070 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.680097 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.680377 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37302: BlockingRecv error: recv got EOF from 127.0.0.1:37302 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 17:50:53.681247 26223 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.678790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.3.222.62:41555)
0928 17:50:53.678826 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.678828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.680764 (+ 1936us) server_negotiation.cc:407] Connection header received
0928 17:50:53.680879 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.680880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.680889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.680916 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.681189 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34242: BlockingRecv error: recv got EOF from 127.0.0.1:34242 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:50:53.683511 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.682856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.3.222.61:45377)
0928 17:50:53.682900 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.682901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.683010 (+ 109us) server_negotiation.cc:407] Connection header received
0928 17:50:53.683108 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.683109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.683126 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.683153 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.683465 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37314: BlockingRecv error: recv got EOF from 127.0.0.1:37314 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 17:50:53.686527 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.685778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.3.222.61:45377)
0928 17:50:53.685817 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.685819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.686067 (+ 248us) server_negotiation.cc:407] Connection header received
0928 17:50:53.686153 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.686154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.686163 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.686189 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.686483 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37316: BlockingRecv error: recv got EOF from 127.0.0.1:37316 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 17:50:53.690294 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.689590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.3.222.61:45377)
0928 17:50:53.689625 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.689627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.689863 (+ 236us) server_negotiation.cc:407] Connection header received
0928 17:50:53.689946 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.689947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.689956 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.689981 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.690254 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37328: BlockingRecv error: recv got EOF from 127.0.0.1:37328 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 17:50:53.696166 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.695410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.3.222.61:45377)
0928 17:50:53.695458 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.695459 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.695695 (+ 236us) server_negotiation.cc:407] Connection header received
0928 17:50:53.695785 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.695786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.695795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.695822 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.696125 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37334: BlockingRecv error: recv got EOF from 127.0.0.1:37334 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 17:50:53.706073 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.705284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.3.222.61:45377)
0928 17:50:53.705323 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.705325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.705576 (+ 251us) server_negotiation.cc:407] Connection header received
0928 17:50:53.705668 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.705670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.705681 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.705708 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.706025 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37344: BlockingRecv error: recv got EOF from 127.0.0.1:37344 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:50:53.724001 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.723302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.3.222.61:45377)
0928 17:50:53.723345 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.723347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.723536 (+ 189us) server_negotiation.cc:407] Connection header received
0928 17:50:53.723583 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.723584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.723595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.723638 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.723946 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37358: BlockingRecv error: recv got EOF from 127.0.0.1:37358 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 17:50:53.757608 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.756942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.3.222.61:45377)
0928 17:50:53.756987 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.756989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.757038 (+ 49us) server_negotiation.cc:407] Connection header received
0928 17:50:53.757121 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.757122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.757131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.757177 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.757546 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37374: BlockingRecv error: recv got EOF from 127.0.0.1:37374 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:50:53.823307 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.822679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.3.222.61:45377)
0928 17:50:53.822773 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.822775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.822791 (+ 16us) server_negotiation.cc:407] Connection header received
0928 17:50:53.822855 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.822856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.822866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.822915 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.823242 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37384: BlockingRecv error: recv got EOF from 127.0.0.1:37384 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 17:50:53.952946 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:53.952303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.3.222.61:45377)
0928 17:50:53.952374 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:53.952376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:53.952426 (+ 50us) server_negotiation.cc:407] Connection header received
0928 17:50:53.952508 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:53.952509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:53.952521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:53.952569 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:53.952885 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37388: BlockingRecv error: recv got EOF from 127.0.0.1:37388 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 17:50:54.210916 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:54.210333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.3.222.61:45377)
0928 17:50:54.210404 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:54.210406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:54.210434 (+ 28us) server_negotiation.cc:407] Connection header received
0928 17:50:54.210462 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:54.210463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:54.210474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:54.210515 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:54.210850 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37392: BlockingRecv error: recv got EOF from 127.0.0.1:37392 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 17:50:54.557140 26388 catalog_manager.cc:1259] Loaded cluster ID: 5657d66f8a1247f49bc08913b4ecd0ce
I20260928 17:50:54.557209 26388 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7: loading cluster ID for follower catalog manager: success
I20260928 17:50:54.557953 26388 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7: acquiring CA information for follower catalog manager: success
I20260928 17:50:54.558218 26388 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4984db952199494e968ae63db5bafde7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:50:54.567770 26402 catalog_manager.cc:1259] Loaded cluster ID: 5657d66f8a1247f49bc08913b4ecd0ce
I20260928 17:50:54.567839 26402 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8: loading cluster ID for follower catalog manager: success
I20260928 17:50:54.568595 26402 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8: acquiring CA information for follower catalog manager: success
I20260928 17:50:54.568836 26402 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 58db5aea4e434c49b0419fe79b7311c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:50:54.726197 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:54.723948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.3.222.61:45377)
0928 17:50:54.724039 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:54.724041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:54.724134 (+ 93us) server_negotiation.cc:407] Connection header received
0928 17:50:54.724162 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:54.724163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:54.724214 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:54.724283 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:54.724400 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.724894 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.725612 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.725793 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.725836 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:54.725946 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:54.726048 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:54.726102 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:54.726116 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 17:50:54.726132 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":43,"threads_started":1}
I20260928 17:50:54.730047 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:54.727917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.3.222.60:42415)
0928 17:50:54.728014 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:54.728016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:54.728024 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:50:54.728062 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:54.728063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:54.728096 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:54.728156 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:54.728229 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.728692 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.729352 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.729551 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.729618 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:54.729715 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:54.729812 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:54.729861 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:54.729945 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 17:50:54.729965 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":53,"threads_started":1}
I20260928 17:50:54.734426 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:54.731646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.3.222.60:42415)
0928 17:50:54.731681 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:54.731682 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:54.731830 (+ 148us) server_negotiation.cc:407] Connection header received
0928 17:50:54.731888 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:54.731889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:54.731898 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:54.731952 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:54.732172 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.732866 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.733694 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.733969 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.734021 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:54.734169 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:54.734201 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:54.734240 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:54.734322 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 17:50:54.734344 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:50:54.734611 26516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:54.731579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.3.222.62:41555)
0928 17:50:54.731684 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:54.731686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:54.731793 (+ 107us) server_negotiation.cc:407] Connection header received
0928 17:50:54.731836 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:54.731837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:54.731886 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:54.731953 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:54.732207 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.732679 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.733740 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.733944 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.733995 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:54.734187 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:54.734295 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:54.734344 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:54.734499 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 17:50:54.734525 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
I20260928 17:50:54.734696 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:54.731562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.3.222.61:45377)
0928 17:50:54.731643 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:54.731644 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:54.731801 (+ 157us) server_negotiation.cc:407] Connection header received
0928 17:50:54.731863 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:54.731865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:54.731876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:54.731929 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:54.732025 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.732725 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.733750 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:54.734026 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:54.734096 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:54.734211 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:54.734239 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:54.734281 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:54.734606 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 17:50:54.734628 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 17:50:54.736153 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 26152
I20260928 17:50:54.740594 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 26227
I20260928 17:50:54.744709 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 26302
2026-09-28T17:50:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2661 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:50:54 dist-test-slave-w3bm krb5kdc[26521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:50:54 dist-test-slave-w3bm krb5kdc[26521](info): set up 2 sockets
Sep 28 17:50:54 dist-test-slave-w3bm krb5kdc[26521](info): commencing operation
krb5kdc: starting...
W20260928 17:50:55.707087 3960 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/test-user.keytab.
Sep 28 17:50:55 dist-test-slave-w3bm krb5kdc[26521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
Entry for principal kudu/127.3.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab.
I20260928 17:50:55.818555 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:44583
--webserver_interface=127.3.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:44583,127.3.222.61:40303,127.3.222.60:46843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:50:55.895366 26534 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:50:55.895563 26534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:55.895587 26534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:55.895610 26534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:55.896695 26534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:55.896733 26534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:55.896750 26534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:55.896767 26534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:55.896781 26534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:55.898217 26534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:44583,127.3.222.61:40303,127.3.222.60:46843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.62.keytab
--principal=kudu/127.3.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.62:44583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:55.898444 26534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:55.898639 26534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:55.903124 26540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:55.903137 26539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:55.903137 26542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:55.903158 26534 server_base.cc:1034] running on GCE node
I20260928 17:50:55.903668 26534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:55.903920 26534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:55.905093 26534 hybrid_clock.cc:648] HybridClock initialized: now 1790617855905079 us; error 28 us; skew 500 ppm
Sep 28 17:50:55 dist-test-slave-w3bm krb5kdc[26521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:55.911789 26534 init.cc:377] Logged in from keytab as kudu/127.3.222.62@KRBTEST.COM (short username kudu)
I20260928 17:50:55.912211 26534 webserver.cc:460] Webserver started at http://127.3.222.62:39563/ using document root <none> and password file <none>
I20260928 17:50:55.912415 26534 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:55.912467 26534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:55.912597 26534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:55.913616 26534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/data/instance:
uuid: "02472595bf8e4f49bbf0a202c8d0205f"
format_stamp: "Formatted at 2026-09-28 17:50:55 on dist-test-slave-w3bm"
server_key: "d82f0b203f3ef96b0872ab0f82c0bb39"
server_key_iv: "402b5ecc29cf940d4f362c42a0867481"
server_key_version: "encryptionkey@0"
I20260928 17:50:55.913935 26534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/wal/instance:
uuid: "02472595bf8e4f49bbf0a202c8d0205f"
format_stamp: "Formatted at 2026-09-28 17:50:55 on dist-test-slave-w3bm"
server_key: "d82f0b203f3ef96b0872ab0f82c0bb39"
server_key_iv: "402b5ecc29cf940d4f362c42a0867481"
server_key_version: "encryptionkey@0"
I20260928 17:50:55.915172 26534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 17:50:55.915855 26549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:55.915988 26534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:55.916069 26534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/wal
uuid: "02472595bf8e4f49bbf0a202c8d0205f"
format_stamp: "Formatted at 2026-09-28 17:50:55 on dist-test-slave-w3bm"
server_key: "d82f0b203f3ef96b0872ab0f82c0bb39"
server_key_iv: "402b5ecc29cf940d4f362c42a0867481"
server_key_version: "encryptionkey@0"
I20260928 17:50:55.916129 26534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:55.927796 26534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:55.929481 26534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:55.929605 26534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:55.933336 26534 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.62:44583
I20260928 17:50:55.933383 26601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.62:44583 every 8 connection(s)
I20260928 17:50:55.933686 26534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/data/info.pb
I20260928 17:50:55.934561 26602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:55.936070 26602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:50:55 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:55.942786 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 26534
I20260928 17:50:55.942884 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-0/wal/instance
I20260928 17:50:55.943077 3960 external_mini_cluster.cc:1428] Setting key f205210a1514d34122588125a8ea9113
I20260928 17:50:55.944727 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:55.936685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:44583 (local address 127.0.0.1:36404)
0928 17:50:55.936860 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:55.936902 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:55.937347 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:55.937607 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:55.937610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:55.937856 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:55.938004 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:55.938008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:55.938954 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:55.938955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:55.939650 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:55.939652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:55.940099 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:55.940100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:55.940159 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:55.940582 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:55.940590 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:55.941900 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:55.943430 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:55.943435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:55.943437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:55.943639 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:55.943836 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:55.943837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:55.943837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:55.943912 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:55.944228 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:55.944232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:55.944288 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 17:50:55.944368 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 17:50:55.944564 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":76,"thread_start_us":84,"threads_started":1}
I20260928 17:50:55.944731 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:55.936726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.3.222.62:44583)
0928 17:50:55.936869 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:55.936879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:55.936990 (+ 111us) server_negotiation.cc:407] Connection header received
0928 17:50:55.937487 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:55.937489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:55.937517 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:55.937596 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:55.938088 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:55.938874 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:55.939718 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:55.940014 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:55.940105 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:55.941986 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:55.941992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:55.941992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:55.941998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:55.943370 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:55.943706 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:55.943709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:55.943709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:55.943785 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:55.943969 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:55.943970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:55.943970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:55.944171 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:55.944237 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:55.944405 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 17:50:55.944510 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260928 17:50:55.945604 26602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:55.945901 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:55.945794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:40303 (local address 127.0.0.1:33486)
0928 17:50:55.945837 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:55.945880 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 17:50:55.946039 26552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
I20260928 17:50:55.946499 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:55.946403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:40303 (local address 127.0.0.1:33492)
0928 17:50:55.946449 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:55.946477 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 17:50:55.946599 26602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.61:40303: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.222.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab.
Entry for principal kudu/127.3.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab.
I20260928 17:50:55.966427 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.61:40303
--webserver_interface=127.3.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab
--principal=kudu/127.3.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:44583,127.3.222.61:40303,127.3.222.60:46843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:50:55.995821 26602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } attempt: 1
I20260928 17:50:55.996315 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:55.996156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:40303 (local address 127.0.0.1:33496)
0928 17:50:55.996226 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:55.996277 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 17:50:55.996704 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:55.996617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:40303 (local address 127.0.0.1:33500)
0928 17:50:55.996657 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:55.996684 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 17:50:55.996805 26602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.61:40303: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
W20260928 17:50:56.043815 26609 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:50:56.044021 26609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:56.044037 26609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:56.044056 26609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:56.045130 26609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:56.045162 26609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:56.045176 26609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:56.045186 26609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:56.045197 26609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:56.046577 26609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:44583,127.3.222.61:40303,127.3.222.60:46843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.61.keytab
--principal=kudu/127.3.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.61:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:56.046756 26609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:56.046916 26609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:56.051559 26617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:56.051553 26614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:56.051625 26615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:56.052134 26609 server_base.cc:1034] running on GCE node
I20260928 17:50:56.052335 26609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:56.052629 26609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:56.053771 26609 hybrid_clock.cc:648] HybridClock initialized: now 1790617856053759 us; error 28 us; skew 500 ppm
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:56.056088 26609 init.cc:377] Logged in from keytab as kudu/127.3.222.61@KRBTEST.COM (short username kudu)
I20260928 17:50:56.056522 26609 webserver.cc:460] Webserver started at http://127.3.222.61:34009/ using document root <none> and password file <none>
I20260928 17:50:56.056708 26609 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:56.056749 26609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:56.056864 26609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:56.057883 26609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/data/instance:
uuid: "8287cee01c1541e0b95aa35f29cf9721"
format_stamp: "Formatted at 2026-09-28 17:50:56 on dist-test-slave-w3bm"
server_key: "1971bad2d489a301886875d8c334292e"
server_key_iv: "8a4b6bdaa5871f3e2960548b2595913e"
server_key_version: "encryptionkey@0"
I20260928 17:50:56.058188 26609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/wal/instance:
uuid: "8287cee01c1541e0b95aa35f29cf9721"
format_stamp: "Formatted at 2026-09-28 17:50:56 on dist-test-slave-w3bm"
server_key: "1971bad2d489a301886875d8c334292e"
server_key_iv: "8a4b6bdaa5871f3e2960548b2595913e"
server_key_version: "encryptionkey@0"
I20260928 17:50:56.059432 26609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:50:56.060141 26624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:56.060284 26609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:50:56.060359 26609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/wal
uuid: "8287cee01c1541e0b95aa35f29cf9721"
format_stamp: "Formatted at 2026-09-28 17:50:56 on dist-test-slave-w3bm"
server_key: "1971bad2d489a301886875d8c334292e"
server_key_iv: "8a4b6bdaa5871f3e2960548b2595913e"
server_key_version: "encryptionkey@0"
I20260928 17:50:56.060410 26609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:56.064931 26602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } attempt: 2
I20260928 17:50:56.065315 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.065203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:40303 (local address 127.0.0.1:33510)
0928 17:50:56.065253 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.065287 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:50:56.065577 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.065506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:40303 (local address 127.0.0.1:33516)
0928 17:50:56.065540 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.065560 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 17:50:56.065668 26602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.61:40303: Network error: Client connection negotiation failed: client connection to 127.3.222.61:40303: connect: Connection refused (error 111)
I20260928 17:50:56.083302 26609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:56.085112 26609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:56.085239 26609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:56.088838 26609 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.61:40303
I20260928 17:50:56.088884 26676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.61:40303 every 8 connection(s)
I20260928 17:50:56.089177 26609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/data/info.pb
I20260928 17:50:56.090255 26677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:56.090550 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 26609
I20260928 17:50:56.090643 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-1/wal/instance
I20260928 17:50:56.090839 3960 external_mini_cluster.cc:1428] Setting key 335b90f8fea3892ba2425ff2e91e0304
I20260928 17:50:56.092382 26677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.222.60@KRBTEST.COM" created.
I20260928 17:50:56.101583 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.093104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.3.222.62:44583)
0928 17:50:56.093153 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.093155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.093352 (+ 197us) server_negotiation.cc:407] Connection header received
0928 17:50:56.093876 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.093877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.093891 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.093972 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.094596 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.095354 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.096361 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.096668 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.096744 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.099014 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.099016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.099016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.099024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.100333 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.100686 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.100687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.100688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.100740 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.100981 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.100982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.100982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.101101 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.101164 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.101401 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.101484 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:50:56.101672 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.093082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:44583 (local address 127.0.0.1:36408)
0928 17:50:56.093241 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.093277 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:56.093778 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:56.094045 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:56.094049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:56.094291 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:56.094505 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.094510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.095441 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.095442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.096274 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.096276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.096770 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.096771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.096879 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.097370 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:56.097378 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:56.098919 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:56.100406 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.100412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.100413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.100624 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.100792 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.100793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.100793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.100887 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.101159 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:56.101161 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:56.101274 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 17:50:56.101365 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 17:50:56.101499 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260928 17:50:56.102241 26677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:50:56.109777 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.102456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:40303 (local address 127.0.0.1:33524)
0928 17:50:56.102502 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.102512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:56.102541 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:56.103160 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:56.103161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:56.103305 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:56.103349 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.103350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.104135 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.104136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.104836 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.104839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.105293 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.105294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.105367 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.105790 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:56.105795 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:56.107077 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:56.108756 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.108757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.108757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.108964 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.109182 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.109183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.109183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.109244 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.109537 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:56.109538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:56.109579 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 17:50:56.109632 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 17:50:56.109709 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 17:50:56.109867 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.102885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.3.222.61:40303)
0928 17:50:56.103027 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.103032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.103051 (+ 19us) server_negotiation.cc:407] Connection header received
0928 17:50:56.103071 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.103072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.103083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.103151 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.103411 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.104060 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.104919 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.105220 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.105277 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.107171 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.107179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.107179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.107185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.108675 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.109059 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.109063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.109063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.109119 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.109299 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.109300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.109300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.109473 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.109537 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.109665 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.109759 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260928 17:50:56.110399 26677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:56.110713 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.110613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40512)
0928 17:50:56.110651 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.110693 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 17:50:56.110842 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
I20260928 17:50:56.111227 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.111120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40528)
0928 17:50:56.111164 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.111187 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 17:50:56.111326 26677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:46843: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab.
Entry for principal kudu/127.3.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab.
I20260928 17:50:56.118233 3960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkw9tqR/build/release/bin/kudu
/tmp/dist-test-taskkw9tqR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.60:46843
--webserver_interface=127.3.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab
--principal=kudu/127.3.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.222.20:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.222.62:44583,127.3.222.61:40303,127.3.222.60:46843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:50:56.156843 26602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } attempt: 3
I20260928 17:50:56.160542 26677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } attempt: 1
I20260928 17:50:56.160914 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.160790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40542)
0928 17:50:56.160849 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.160883 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:50:56.161819 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.161718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40550)
0928 17:50:56.161760 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.161791 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 17:50:56.161916 26677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:46843: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
I20260928 17:50:56.164428 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.157247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:40303 (local address 127.0.0.1:33536)
0928 17:50:56.157313 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.157327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:56.157388 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:56.157522 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:56.157523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:56.157679 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:56.157743 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.157745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.158677 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.158678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.159270 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.159273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.159775 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.159776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.159842 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.160336 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:56.160341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:56.161778 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:56.163563 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.163564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.163565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.163769 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.163945 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.163945 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.163946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.163990 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.164214 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:56.164214 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:56.164252 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 17:50:56.164296 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:50:56.164366 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 17:50:56.164466 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.157305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.3.222.61:40303)
0928 17:50:56.157373 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.157375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.157389 (+ 14us) server_negotiation.cc:407] Connection header received
0928 17:50:56.157439 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.157440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.157452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.157512 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.157804 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.158579 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.159343 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.159688 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.159764 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.161874 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.161875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.161876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.161883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.163458 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.163835 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.163836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.163836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.163894 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.164035 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.164035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.164036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.164160 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.164222 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.164319 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.164389 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:50:56.164919 26602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:50:56.165239 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.165151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40556)
0928 17:50:56.165192 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.165218 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 17:50:56.165879 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.165808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40558)
0928 17:50:56.165844 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.165865 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 17:50:56.165961 26602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:46843: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
I20260928 17:50:56.197141 26602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } attempt: 1
I20260928 17:50:56.197592 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.197460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40560)
0928 17:50:56.197527 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.197558 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 17:50:56.197955 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.197862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40576)
0928 17:50:56.197899 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.197936 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 17:50:56.198050 26602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:46843: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
W20260928 17:50:56.199592 26684 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:50:56.199775 26684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:50:56.199791 26684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:50:56.199815 26684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:50:56.200897 26684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:50:56.200932 26684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:50:56.200945 26684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:50:56.200956 26684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:50:56.200973 26684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:50:56.202358 26684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.222.20:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.222.62:44583,127.3.222.61:40303,127.3.222.60:46843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/krb5kdc/kudu_127.3.222.60.keytab
--principal=kudu/127.3.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.222.60:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:50:56.202535 26684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:50:56.202698 26684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:50:56.207345 26692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:50:56.207530 26690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:56.207466 26684 server_base.cc:1034] running on GCE node
W20260928 17:50:56.207448 26689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:50:56.207835 26684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:50:56.208109 26684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:50:56.209281 26684 hybrid_clock.cc:648] HybridClock initialized: now 1790617856209261 us; error 31 us; skew 500 ppm
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:50:56.211678 26684 init.cc:377] Logged in from keytab as kudu/127.3.222.60@KRBTEST.COM (short username kudu)
I20260928 17:50:56.212071 26684 webserver.cc:460] Webserver started at http://127.3.222.60:34575/ using document root <none> and password file <none>
I20260928 17:50:56.212268 26684 fs_manager.cc:359] Metadata directory not provided
I20260928 17:50:56.212325 26684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:50:56.212458 26684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:50:56.213464 26684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/data/instance:
uuid: "a921d777b7e648f2a4580ce4e31ce7ee"
format_stamp: "Formatted at 2026-09-28 17:50:56 on dist-test-slave-w3bm"
server_key: "de15e7398bfb097557c455cd3b85787c"
server_key_iv: "aaf79478a5927d6a98a9a7f7e47802af"
server_key_version: "encryptionkey@0"
I20260928 17:50:56.213785 26684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/wal/instance:
uuid: "a921d777b7e648f2a4580ce4e31ce7ee"
format_stamp: "Formatted at 2026-09-28 17:50:56 on dist-test-slave-w3bm"
server_key: "de15e7398bfb097557c455cd3b85787c"
server_key_iv: "aaf79478a5927d6a98a9a7f7e47802af"
server_key_version: "encryptionkey@0"
I20260928 17:50:56.215023 26684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:50:56.215752 26699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:56.215898 26684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:50:56.215977 26684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/data,/tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/wal
uuid: "a921d777b7e648f2a4580ce4e31ce7ee"
format_stamp: "Formatted at 2026-09-28 17:50:56 on dist-test-slave-w3bm"
server_key: "de15e7398bfb097557c455cd3b85787c"
server_key_iv: "aaf79478a5927d6a98a9a7f7e47802af"
server_key_version: "encryptionkey@0"
I20260928 17:50:56.216044 26684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:50:56.227885 26684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:50:56.229823 26684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:50:56.229965 26684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:50:56.230047 26677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } attempt: 2
I20260928 17:50:56.230453 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.230304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40588)
0928 17:50:56.230369 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.230413 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 17:50:56.230916 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.230772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40596)
0928 17:50:56.230833 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.230866 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 17:50:56.231055 26677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.222.60:46843: Network error: Client connection negotiation failed: client connection to 127.3.222.60:46843: connect: Connection refused (error 111)
I20260928 17:50:56.233860 26684 rpc_server.cc:307] RPC server started. Bound to: 127.3.222.60:46843
I20260928 17:50:56.233908 26751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.222.60:46843 every 8 connection(s)
I20260928 17:50:56.234206 26684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/data/info.pb
I20260928 17:50:56.235157 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:50:56.236670 26752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:56.242745 3960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkw9tqR/build/release/bin/kudu as pid 26684
I20260928 17:50:56.242825 3960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkw9tqR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617599653609-3960-0/minicluster-data/master-2/wal/instance
I20260928 17:50:56.243000 3960 external_mini_cluster.cc:1428] Setting key f43fcd13a1d1235f7dee7fe711af5256
I20260928 17:50:56.245085 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.237280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.3.222.62:44583)
0928 17:50:56.237331 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.237333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.237501 (+ 168us) server_negotiation.cc:407] Connection header received
0928 17:50:56.237990 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.237992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.238002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.238060 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.238650 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.239149 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.240168 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.240357 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.240412 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.242356 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.242358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.242358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.242365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.243885 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.244214 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.244215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.244215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.244274 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.244501 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.244501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.244502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.244601 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.244648 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.244952 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.245021 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:50:56.245448 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.237251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.62:44583 (local address 127.0.0.1:36416)
0928 17:50:56.237393 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.237433 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:56.237861 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:56.238121 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:56.238124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:56.238346 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:56.238562 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.238567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.239277 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.239278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.240087 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.240090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.240428 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.240428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.240527 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.240981 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:56.240989 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:56.242277 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:56.243946 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.243949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.243950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.244152 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.244330 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.244330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.244330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.244444 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.244655 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:56.244660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:56.244800 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 17:50:56.244904 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 17:50:56.245247 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":188,"thread_start_us":57,"threads_started":1}
I20260928 17:50:56.245972 26752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
I20260928 17:50:56.251995 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.243354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.3.222.60:46843)
0928 17:50:56.243517 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.243522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.243572 (+ 50us) server_negotiation.cc:407] Connection header received
0928 17:50:56.243644 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.243649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.243662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.243735 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.244361 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.245314 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.246172 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.246563 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.246620 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.249115 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.249123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.249123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.249129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.250834 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.251175 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.251178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.251178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.251254 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.251425 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.251426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.251426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.251592 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.251710 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.251791 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.251897 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260928 17:50:56.252889 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.246194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.61:40303 (local address 127.0.0.1:33544)
0928 17:50:56.246225 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.246235 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:56.246266 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:56.246403 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:56.246404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:56.246505 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:56.246560 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.246562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.247290 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.247292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.248082 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.248085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.248529 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.248530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.248588 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.248929 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:56.248933 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:56.250221 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:56.251683 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.251684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.251684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.251913 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.252269 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.252270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.252270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.252319 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.252610 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:56.252611 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:56.252653 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 17:50:56.252719 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 17:50:56.252802 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 17:50:56.252939 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.246259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.3.222.61:40303)
0928 17:50:56.246302 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.246303 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.246310 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:50:56.246332 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.246332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.246341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.246402 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.246630 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.247230 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.248146 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.248451 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.248521 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.250308 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.250309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.250309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.250316 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.251614 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.251980 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.251981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.251981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.252035 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.252384 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.252385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.252385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.252531 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.252606 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.252785 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.252866 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:50:56.253221 26752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.60@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.62@KRBTEST.COM
I20260928 17:50:56.260152 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.253455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40618)
0928 17:50:56.253481 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.253488 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:56.253518 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:56.253618 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:56.253619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:56.253715 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:56.253747 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.253749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.254321 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.254322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.254785 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.254786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.255139 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.255139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.255183 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.255641 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:56.255645 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:56.257549 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:56.259189 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.259190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.259190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.259419 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.259598 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.259598 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.259598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.259643 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.259866 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:56.259866 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:56.259898 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 17:50:56.259957 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 17:50:56.260042 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260928 17:50:56.260214 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.253485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.3.222.60:46843)
0928 17:50:56.253516 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.253518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.253525 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:50:56.253548 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.253548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.253557 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.253612 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.253815 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.254255 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.254850 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.255076 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.255132 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.257631 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.257632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.257633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.257639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.259106 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.259497 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.259498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.259499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.259543 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.259697 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.259698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.259698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.259799 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.259847 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.259979 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.260102 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":17}
I20260928 17:50:56.260772 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.252790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.3.222.62:44583)
0928 17:50:56.252830 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.252832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.253018 (+ 186us) server_negotiation.cc:407] Connection header received
0928 17:50:56.253096 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.253098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.253109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.253158 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.253763 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.254230 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.254878 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.255054 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.255099 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.257724 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.257726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.257726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.257733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.259379 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.259706 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.259708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.259708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.259778 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.260019 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.260020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.260020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.260191 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.260349 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.260583 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.260688 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:50:56.262552 26752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee: Bootstrap starting.
I20260928 17:50:56.263513 26752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:56.263784 26752 log.cc:824] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:56.264518 26752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee: No bootstrap required, opened a new log
I20260928 17:50:56.266189 26752 raft_consensus.cc:348] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } }
I20260928 17:50:56.266309 26752 raft_consensus.cc:729] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a921d777b7e648f2a4580ce4e31ce7ee, State: Initialized, Role: FOLLOWER
I20260928 17:50:56.266383 26752 consensus_queue.cc:260] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } }
I20260928 17:50:56.266695 26752 sys_catalog.cc:564] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:56.266652 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.266753 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.222.61@KRBTEST.COM
W20260928 17:50:56.268435 26769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:56.268671 26769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:56.270048 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.261713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.3.222.61:40303)
0928 17:50:56.262565 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.262566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.262589 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:50:56.262610 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.262611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.262621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.262674 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.263261 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.263948 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.265012 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.265297 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.265369 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.267595 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.267597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.267597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.267604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.269058 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.269331 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.269332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.269333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.269387 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.269547 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.269548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.269549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.269693 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.269853 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.269893 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.269965 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828}
I20260928 17:50:56.271636 3960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:50:56.269476 26684 master_runner.cc:186] Error getting master registration for 127.3.222.62:44583: OK, instance_id { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" instance_seqno: 1790617855932292 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:50:56.271752 26684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:50:56.272495 26576 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "02472595bf8e4f49bbf0a202c8d0205f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36422
I20260928 17:50:56.273204 26602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } attempt: 2
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.62@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
I20260928 17:50:56.280054 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.273500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.3.222.60:46843)
0928 17:50:56.273537 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.273538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.273571 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:50:56.273590 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.273590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.273599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.273642 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.273909 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.274614 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.275539 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.275824 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.275895 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.277746 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.277748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.277748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.277755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.278930 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.279318 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.279319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.279319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.279372 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.279554 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.279555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.279555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.279703 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.279761 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.279897 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.279966 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:50:56.280054 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.273438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40628)
0928 17:50:56.273484 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.273495 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:56.273531 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:56.273653 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:56.273654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:56.273806 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:56.273847 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.273848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.274684 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.274684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.275471 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.275472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.275890 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.275891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.275940 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.276328 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:56.276333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:56.277648 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:56.278995 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.278997 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.278997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.279253 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.279431 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.279432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.279433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.279501 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.279761 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:56.279762 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:56.279807 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 17:50:56.279872 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 17:50:56.279959 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 17:50:56.281459 26602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f: Bootstrap starting.
I20260928 17:50:56.282151 26602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:56.282411 26602 log.cc:824] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:56.283077 26602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f: No bootstrap required, opened a new log
I20260928 17:50:56.284360 26602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } }
I20260928 17:50:56.284492 26602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02472595bf8e4f49bbf0a202c8d0205f, State: Initialized, Role: FOLLOWER
I20260928 17:50:56.284567 26602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } }
I20260928 17:50:56.284799 26772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.284862 26602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:56.284885 26576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 17:50:56.284921 26772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:56.284979 26576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:56.285802 26576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } }
I20260928 17:50:56.286461 26726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02472595bf8e4f49bbf0a202c8d0205f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a921d777b7e648f2a4580ce4e31ce7ee"
I20260928 17:50:56.286609 26726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:56.286109 26576 leader_election.cc:288] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [CANDIDATE]: Term 1 election: Requested vote from peers 8287cee01c1541e0b95aa35f29cf9721 (127.3.222.61:40303), a921d777b7e648f2a4580ce4e31ce7ee (127.3.222.60:46843)
I20260928 17:50:56.287480 26726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02472595bf8e4f49bbf0a202c8d0205f in term 1.
I20260928 17:50:56.287714 26550 leader_election.cc:302] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02472595bf8e4f49bbf0a202c8d0205f, a921d777b7e648f2a4580ce4e31ce7ee; no voters:
I20260928 17:50:56.287817 26772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:50:56.287937 26772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [term 1 LEADER]: Becoming Leader. State: Replica: 02472595bf8e4f49bbf0a202c8d0205f, State: Running, Role: LEADER
I20260928 17:50:56.288009 26772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } }
I20260928 17:50:56.288406 26772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02472595bf8e4f49bbf0a202c8d0205f. Latest consensus state: current_term: 1 leader_uuid: "02472595bf8e4f49bbf0a202c8d0205f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.288468 26772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [sys.catalog]: This master's current role is: LEADER
W20260928 17:50:56.286793 26783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:56.289094 26783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:56.289341 26534 master_runner.cc:186] Error getting master registration for 127.3.222.61:40303: OK, instance_id { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" instance_seqno: 1790617856087837 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:50:56.291626 26534 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:50:56.296271 26791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:50:56.296510 26791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:50:56.305083 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.295475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.3.222.62:44583)
0928 17:50:56.295511 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.295512 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.297380 (+ 1868us) server_negotiation.cc:407] Connection header received
0928 17:50:56.297516 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.297517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.297527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.297555 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.298251 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.298971 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.300124 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.300415 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.300450 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.301850 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.301851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.301852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.301858 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.303939 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.304291 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.304292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.304293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.304349 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.304539 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.304540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.304540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.304653 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.304725 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.304924 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.304996 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:50:56.305157 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.288792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.3.222.60:46843)
0928 17:50:56.288827 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.288829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.295722 (+ 6893us) server_negotiation.cc:407] Connection header received
0928 17:50:56.295863 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.295864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.295887 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.295916 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.297200 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.297953 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.299483 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.299795 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.299834 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.302000 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.302001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.302002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.302020 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.303582 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.304145 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.304146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.304147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.304203 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.304469 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.304470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.304471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.304609 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.304691 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.304990 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.305066 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:50:56.306576 26726 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [term 1 FOLLOWER]: Refusing update from remote peer 02472595bf8e4f49bbf0a202c8d0205f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:56.306975 26785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:56.308344 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02472595bf8e4f49bbf0a202c8d0205f. Latest consensus state: current_term: 1 leader_uuid: "02472595bf8e4f49bbf0a202c8d0205f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.308422 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:56.309639 26785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02472595bf8e4f49bbf0a202c8d0205f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.309724 26785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:56.310168 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02472595bf8e4f49bbf0a202c8d0205f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.310240 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:56.310539 26795 mvcc.cc:205] Tried to move back new op lower bound from 7334370739429142528 to 7334370739356364800. Current Snapshot: MvccSnapshot[applied={T|T < 7334370739429142528}]
I20260928 17:50:56.311262 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.295689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.3.222.61:40303)
0928 17:50:56.295733 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.295734 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.303341 (+ 7607us) server_negotiation.cc:407] Connection header received
0928 17:50:56.303451 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.303453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.303463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.303493 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.304090 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.304747 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.305669 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.305920 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.305960 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.308099 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.308101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.308101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.308120 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.309748 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.310283 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.310285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.310285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.310338 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.310594 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.310595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.310595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.310729 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.310792 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.311083 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.311156 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:50:56.311537 26796 mvcc.cc:205] Tried to move back new op lower bound from 7334370739429142528 to 7334370739356364800. Current Snapshot: MvccSnapshot[applied={T|T < 7334370739429142528}]
I20260928 17:50:56.315620 26791 catalog_manager.cc:1347] Generated new cluster ID: 5d9ede901ff24325996a86f23f719e9e
I20260928 17:50:56.315671 26791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:50:56.322204 26677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } attempt: 3
Sep 28 17:50:56 dist-test-slave-w3bm krb5kdc[26521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.222.61@KRBTEST.COM for kudu/127.3.222.60@KRBTEST.COM
I20260928 17:50:56.330174 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.322534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.3.222.60:46843)
0928 17:50:56.322586 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.322589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.322687 (+ 98us) server_negotiation.cc:407] Connection header received
0928 17:50:56.322708 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.322709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.322718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.322757 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.323105 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.323870 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.324807 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.325082 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.325134 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.327721 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:50:56.327722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:50:56.327723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:50:56.327729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:50:56.329143 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.329479 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.329480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.329480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.329530 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:50:56.329697 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:50:56.329698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:50:56.329698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:50:56.329819 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:50:56.329871 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:56.330006 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 17:50:56.330076 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:50:56.330173 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:56.322465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.222.60:46843 (local address 127.0.0.1:40656)
0928 17:50:56.322581 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 17:50:56.322593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:50:56.322638 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:50:56.322796 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:50:56.322797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:50:56.322968 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:50:56.323028 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.323030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.323959 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.323960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.324730 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:50:56.324731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:50:56.325183 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:50:56.325184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:50:56.325233 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:56.326226 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:50:56.326231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:50:56.327646 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:50:56.329204 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.329205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.329205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.329404 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.329582 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:50:56.329583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:50:56.329583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:50:56.329636 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:50:56.329881 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:50:56.329882 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:50:56.329924 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 17:50:56.329987 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 17:50:56.330076 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 17:50:56.331890 26677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721: Bootstrap starting.
I20260928 17:50:56.332538 26677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721: Neither blocks nor log segments found. Creating new log.
I20260928 17:50:56.332803 26677 log.cc:824] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721: Log is configured to *not* fsync() on all Append() calls
I20260928 17:50:56.339934 26677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721: No bootstrap required, opened a new log
I20260928 17:50:56.341758 26677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } }
I20260928 17:50:56.341964 26677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8287cee01c1541e0b95aa35f29cf9721, State: Initialized, Role: FOLLOWER
I20260928 17:50:56.342103 26677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } }
I20260928 17:50:56.342442 26677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:50:56.344233 26650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:50:56.345093 26650 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [term 1 FOLLOWER]: Refusing update from remote peer 02472595bf8e4f49bbf0a202c8d0205f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:50:56.345449 26772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:50:56.345741 26651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02472595bf8e4f49bbf0a202c8d0205f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8287cee01c1541e0b95aa35f29cf9721"
I20260928 17:50:56.346429 26651 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02472595bf8e4f49bbf0a202c8d0205f in term 1.
W20260928 17:50:56.349002 26816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:50:56.349112 26816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:50:56.349243 26810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02472595bf8e4f49bbf0a202c8d0205f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.349341 26810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:56.349411 26810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02472595bf8e4f49bbf0a202c8d0205f. Latest consensus state: current_term: 1 leader_uuid: "02472595bf8e4f49bbf0a202c8d0205f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.349488 26810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:56.349543 26810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02472595bf8e4f49bbf0a202c8d0205f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.349615 26810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:50:56.351150 26808 mvcc.cc:205] Tried to move back new op lower bound from 7334370739429142528 to 7334370739356364800. Current Snapshot: MvccSnapshot[applied={T|T < 7334370739429142528}]
I20260928 17:50:56.365070 26772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02472595bf8e4f49bbf0a202c8d0205f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02472595bf8e4f49bbf0a202c8d0205f" member_type: VOTER last_known_addr { host: "127.3.222.62" port: 44583 } } peers { permanent_uuid: "8287cee01c1541e0b95aa35f29cf9721" member_type: VOTER last_known_addr { host: "127.3.222.61" port: 40303 } } peers { permanent_uuid: "a921d777b7e648f2a4580ce4e31ce7ee" member_type: VOTER last_known_addr { host: "127.3.222.60" port: 46843 } } }
I20260928 17:50:56.365195 26772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f [sys.catalog]: This master's current role is: LEADER
I20260928 17:50:56.366276 26791 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:50:56.367100 26791 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:50:56.378558 26791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02472595bf8e4f49bbf0a202c8d0205f: Generated new TSK 0
I20260928 17:50:56.378752 26791 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 17:50:56.407912 26756 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.406049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.3.222.60:46843)
0928 17:50:56.406102 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.406104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.407323 (+ 1219us) server_negotiation.cc:407] Connection header received
0928 17:50:56.407412 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.407413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.407422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.407460 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.407840 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40662: BlockingRecv error: recv got EOF from 127.0.0.1:40662 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 17:50:56.408484 26605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.406185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.3.222.62:44583)
0928 17:50:56.406227 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.406229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.406893 (+ 664us) server_negotiation.cc:407] Connection header received
0928 17:50:56.406998 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.407000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.407010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.407040 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.408408 (+ 1368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36436: BlockingRecv error: recv got EOF from 127.0.0.1:36436 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 17:50:56.409423 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.406542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.3.222.61:40303)
0928 17:50:56.406580 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.406582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.408893 (+ 2311us) server_negotiation.cc:407] Connection header received
0928 17:50:56.409032 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.409033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.409043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.409071 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.409366 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33578: BlockingRecv error: recv got EOF from 127.0.0.1:33578 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 17:50:56.411767 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.410967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.3.222.61:40303)
0928 17:50:56.411000 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.411001 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.411197 (+ 196us) server_negotiation.cc:407] Connection header received
0928 17:50:56.411334 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.411335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.411352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.411379 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.411723 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33594: BlockingRecv error: recv got EOF from 127.0.0.1:33594 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 17:50:56.414662 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.413963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.3.222.61:40303)
0928 17:50:56.413993 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.413995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.414211 (+ 216us) server_negotiation.cc:407] Connection header received
0928 17:50:56.414295 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.414296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.414305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.414331 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.414620 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33598: BlockingRecv error: recv got EOF from 127.0.0.1:33598 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 17:50:56.419641 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.418826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.3.222.61:40303)
0928 17:50:56.418885 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.418886 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.419113 (+ 227us) server_negotiation.cc:407] Connection header received
0928 17:50:56.419199 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.419200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.419238 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.419267 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.419574 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33610: BlockingRecv error: recv got EOF from 127.0.0.1:33610 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 17:50:56.425585 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.424867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.3.222.61:40303)
0928 17:50:56.424898 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.424899 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.425127 (+ 228us) server_negotiation.cc:407] Connection header received
0928 17:50:56.425208 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.425209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.425219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.425244 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.425542 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33626: BlockingRecv error: recv got EOF from 127.0.0.1:33626 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 17:50:56.435652 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.434782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.3.222.61:40303)
0928 17:50:56.434823 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.434825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.435078 (+ 253us) server_negotiation.cc:407] Connection header received
0928 17:50:56.435173 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.435174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.435185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.435235 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.435597 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33628: BlockingRecv error: recv got EOF from 127.0.0.1:33628 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 17:50:56.454025 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.453015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.3.222.61:40303)
0928 17:50:56.453059 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.453061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.453347 (+ 286us) server_negotiation.cc:407] Connection header received
0928 17:50:56.453465 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.453466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.453480 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.453568 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.453953 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33640: BlockingRecv error: recv got EOF from 127.0.0.1:33640 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 17:50:56.487787 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.487036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.3.222.61:40303)
0928 17:50:56.487088 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.487090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.487168 (+ 78us) server_negotiation.cc:407] Connection header received
0928 17:50:56.487272 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.487273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.487285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.487340 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.487724 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33642: BlockingRecv error: recv got EOF from 127.0.0.1:33642 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 17:50:56.553447 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.552825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.3.222.61:40303)
0928 17:50:56.552875 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.552877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.552972 (+ 95us) server_negotiation.cc:407] Connection header received
0928 17:50:56.553041 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.553042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.553052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.553112 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.553385 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33656: BlockingRecv error: recv got EOF from 127.0.0.1:33656 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 17:50:56.683265 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.682572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.3.222.61:40303)
0928 17:50:56.682671 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.682673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.682794 (+ 121us) server_negotiation.cc:407] Connection header received
0928 17:50:56.682825 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.682827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.682838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.682882 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.683180 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33670: BlockingRecv error: recv got EOF from 127.0.0.1:33670 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 17:50:56.940889 26682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:50:56.940327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.3.222.61:40303)
0928 17:50:56.940377 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:56.940379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:56.940389 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:50:56.940438 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:56.940439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:56.940448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:56.940495 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:50:56.940832 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33682: BlockingRecv error: recv got EOF from 127.0.0.1:33682 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:50:57.269124 26769 catalog_manager.cc:1259] Loaded cluster ID: 5d9ede901ff24325996a86f23f719e9e
I20260928 17:50:57.269198 26769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee: loading cluster ID for follower catalog manager: success
I20260928 17:50:57.270002 26769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee: acquiring CA information for follower catalog manager: success
I20260928 17:50:57.270270 26769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a921d777b7e648f2a4580ce4e31ce7ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:50:57.349556 26816 catalog_manager.cc:1259] Loaded cluster ID: 5d9ede901ff24325996a86f23f719e9e
I20260928 17:50:57.349634 26816 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721: loading cluster ID for follower catalog manager: success
I20260928 17:50:57.350450 26816 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721: acquiring CA information for follower catalog manager: success
I20260928 17:50:57.350728 26816 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8287cee01c1541e0b95aa35f29cf9721: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:50:57.456406 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:57.453923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.3.222.61:40303)
0928 17:50:57.454043 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:57.454045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:57.454070 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:50:57.454137 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:57.454138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:57.454189 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:57.454259 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:57.454421 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.454926 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.455668 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.455863 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.455916 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:57.456091 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:57.456207 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:57.456289 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:57.456301 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 17:50:57.456319 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260928 17:50:57.460165 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:57.458079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.3.222.60:46843)
0928 17:50:57.458191 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:57.458193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:57.458217 (+ 24us) server_negotiation.cc:407] Connection header received
0928 17:50:57.458250 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:57.458251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:57.458285 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:57.458339 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:57.458412 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.458907 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.459567 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.459754 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.459795 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:57.459886 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:57.459983 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:57.460035 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:57.460068 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 17:50:57.460084 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260928 17:50:57.464618 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:57.461738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.3.222.61:40303)
0928 17:50:57.461784 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:57.461786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:57.461901 (+ 115us) server_negotiation.cc:407] Connection header received
0928 17:50:57.461978 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:57.461979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:57.461991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:57.462031 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:57.462143 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.462925 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.463863 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.464142 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.464203 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:57.464355 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:57.464389 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:57.464434 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:57.464522 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 17:50:57.464545 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:50:57.464676 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:57.461724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.3.222.60:46843)
0928 17:50:57.461763 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:57.461765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:57.461896 (+ 131us) server_negotiation.cc:407] Connection header received
0928 17:50:57.462079 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:57.462080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:57.462090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:57.462155 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:57.462299 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.462863 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.463982 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.464255 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.464309 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:57.464454 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:57.464487 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:57.464531 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:57.464580 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 17:50:57.464599 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:50:57.465023 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:50:57.461605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.3.222.62:44583)
0928 17:50:57.461712 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 17:50:57.461714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:50:57.461844 (+ 130us) server_negotiation.cc:407] Connection header received
0928 17:50:57.461939 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:50:57.461940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:50:57.461992 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:50:57.462064 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:50:57.462208 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.462983 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.464039 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:50:57.464279 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:50:57.464334 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:50:57.464536 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:50:57.464728 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:50:57.464781 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:50:57.464925 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 17:50:57.464948 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260928 17:50:57.466271 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 26534
I20260928 17:50:57.470789 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 26609
I20260928 17:50:57.475169 3960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkw9tqR/build/release/bin/kudu with pid 26684
2026-09-28T17:50:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2730 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5392 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (257827 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST